'AVRO - JSON Enconding of messages in Confluent Python SerializingProducer vs. Confluent Rest Proxy with UNIONS

attached an example AVRO-Schema

{
  "type": "record",
  "name": "DummySampleAvroValue",
  "namespace": "de.company.dummydomain",
  "fields": [
    {
      "name": "ID",
      "type": "int"
    },
    {
      "name": "NAME",
      "type": [
        "null",
        "string"
      ]
    },
    {
      "name": "STATE",
      "type": "int"
    },
    {
      "name": "TIMESTAMP",
      "type": [
        "null",
        "string"
      ]
    }
  ]
}

Regarding the section "JSON Encoding" of the official AVRO-Specs - see: https://avro.apache.org/docs/current/spec.html#json_encoding - a JSON Message which validates against the above AVRO-Schema should look like the following because of the UNION-Types used:

{
   "ID":1,
   "NAME":{
      "string":"Kafka"
   },
   "STATE":-1,
   "TIMESTAMP":{
      "string":"2022-04-28T10:57:03.048413"
   }
}

When producing this message via Confluent Rest Proxy (AVRO), everything works fine, the data is accepted, validated and present in Kafka.

When using the "SearializingProducer" from the confluent_kafka Python Package, the example message is not accepted and only "regular" JSON works, e. g.:

{
   "ID":1,
   "NAME":"Kafka",
   "STATE":-1,
   "TIMESTAMP":"2022-04-28T10:57:03.048413"
}

Is this intended behaviour or am I doing something wrong? Can I tell the SerializingProducer to accept this encoding?

I need to hold open both ways to produce messages but the sending system can/want´s only to provide one of the above Payloads. Is there a way to use both with the same payload?

Thanks in advance.

Best regards



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source