schematics-proto3

Built upon Schematics - Python Data Structures for Humans™, schematics-proto3 brings the awesome features of Schematics to Protobuf 3 world.

Note

Library is currently in WORK IN PROGRESS state.

What is implemented and tested:

  1. Loading Protobuf 3 messages to Model instances.

    • for most of the Protobuf 3 types, including wrappers, repeated and oneof fields

  2. Validation and structured error messages.

To be done:

  1. Serializing Model instances to Protobuf 3 messages.

  2. Enum type.

  3. Make the library more user-friendly.

  4. Schematics “roles”.

Motivation

As good and widely supported as it is, Protobuf 3 still has some quirks which can make working with it painful and repetitive. Especially, building complex gRPC services might reveal a number of deficiencies in available tooling.

schematics-proto3 aims to address this problem, in particular:

  • [#359] default values and testing if a field is set in v3

    There is a workaround for this, schematics-proto3 incorporates wrapper types to hide nested messages underneath.

  • no proper data handling library

    Comparing to Serializers in Django Rest Framework or Marshmallow, there seems to be no full fledged serialization / validation / deserialization library for Protobuf 3. Thanks to Schematics, schematics-proto3 is able to provide:

    • declarative models

    • custom validation functions

    • structured error messages (currently only as Python dict)

Example

Let’s take Schematics example and modify it to work with Protobuf.

We have a following Protobuf message (and person_pb2 Python module).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
syntax = "proto3";

import "google/protobuf/wrappers.proto";

package example;

message Person {
    google.protobuf.StringValue name = 1;
    google.protobuf.StringValue website = 2;
}

And reflect above message in Model class.

1
2
3
4
5
6
7
8
9
from schematics_proto3 import Model
from schematics_proto3 import types as pbtypes

import person_pb2 as pb2


class PersonModel(Model, protobuf_message=pb2.Person):
    name = pbtypes.StringWrapperType()
    website = pbtypes.StringWrapperType()

Let’s load some data.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
msg = pb2.Person()
msg.name.value = 'Jon Doe'
msg.website.value = 'https://example.com'

model = PersonModel.load_protobuf(msg)
model.validate()

assert model.name == 'Jon Doe'
assert model.website == 'https://example.com'

assert model.to_native() == {'name': 'Jon Doe', 'website': 'https://example.com'}

Not setting a field will give you an Unset.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
from schematics_proto3.unset import Unset

msg = pb2.Person()
msg.name.value = 'Jon Doe'

model = PersonModel.load_protobuf(msg)
model.validate()

assert model.name == 'Jon Doe'
assert model.website is Unset

Installation

Currently possible only from repository. Let me know if you want to use it, I will create a PyPI project.