Send heavy data through protobuf. Custom field

Question

I'm developing the API for the application using protobuf and grpc.

I need to send the data with the arbitrary size. Sometimes it is small, sometimes huge. For example Nympy array. If the size is small I want to send it through protobuf, if the size if huge I want to dump data into file and send the filepath to this file through protobuf.

To do so I've created a following .proto messages:

message NumpyTroughProtobuf {
    repeated int32 shape = 1;
    repeated float array = 2;
}
message NumpyTroughfile {
    string filepath = 1;
}
message NumpyTrough {
  google.protobuf.Any data = 1;
}

The logic is simple: If the size is big I use data as NumpyTroughfile or if small data as NumpyTroughProtobuf.

Problem (what I want to avoid):

The mechanism of data transformation is the part of my app.

In the current approach I have to check and covert the data before I create the message NumpyTrough. So I have to add some logic into my application which will care of data check and cast. The same I have to do for any language which I use (for example if I send massages from Python to C++).

What I want to do:

The mechanism of data transformation is the part of customized protobuf.

I want to hide the data transformation. I want that my app to send a pure Numpy array into NumpyTrough.data field. All data transformation should be hided.

So I want that the logic of data transformation be the part of custom Protobuf field, not the part of my application.

Which meant that I would like to create a custom type of the field. I just implement the behavior of this filed (marshal/unmarshal) for any languages which I use. Then I can just directly send Numpy data into this custom field and this field will decide how to proceed: turn the data in into file or via other method, send trough Protobuf and restore on the receiver side.

Somethig like this https://github.com/gogo/protobuf/blob/master/custom_types.md but it seems this is not a part of protobuf ecosystem.

Answer 1

Protobuf only defines schema.

You can't add logic to a protobuf definition.

Protobug Any represents arbitrary binary data and so -- somewhere -- you'll need to explain to your users what it represents in order that they can ship data in the correct format to your service.

You get to decide how to distribute the processing of the data:

• Either partly client-side functionality that performs preprocessing of the data and ships the output (either as structured data using non-Any types or, if still necessary as Any).
• Or partly server-side that receives entirely unprocessed client-side data shipped through Any
• Or some combination of the two

NOTE You may want to consider shipping the data regardless of size as file references to simplify your implementation. You're correct to bias protobuf to smaller message sizes but, depending on the file size distribution, does it make sense to complicate your implementation with 2 paths?