How to update Avro schema with a reference to another schema on a Kafka topic?

Question

What is the correct way to update an Avro schema on a Kafka topic, if this schema is used as a reference in another schema?

For example, let's say we have two Kafka topics: one uses Avro schema User {"type" : "record", "namespace" : "test", "name" : "User", "fields" : [{"name": "username", "type": "string"}]} and the second one UserAction {"type" : "record", "namespace" : "test", "name" : "UserAction", "fields" : [{"name": "action", "type": "string"}, {"name": "user", "type": "test.User"}]}.

Then I want to add an additional field to the User - a "surname", so it will look like this: ... "fields" : [{"name": "username", "type": "string"}, {"name": "surname", "type": ["string", "null"], "default": null}], null to make this change a compatible one. To do this I can change the Avro schema file, regenerate POJOs using Maven schema plugin, and then if I'll send a message to the first topic with a KafkaTemplate, the schema will be updated and the new field will be visible on the topic.

The issue is that if I'll send a message with UserAction to the second topic, it would still refer to the old User schema, without the "surname" field, even though POJOs will see it correctly. And because of this any "surname" sent won't be stored in the topic and would be received as a null in Consumer.

Is there any way to force update UserAction schema on the second topic to refer to the new User schema?

Answer 1

While the Confluent Schema Registry allows for references at registration time, I don't think it'll dynamically update as you change only one model.

Instead, you can define a schema "monorepo" where you package and register your schema changes together.

For example, in Avro IDL, you could define one file

record User {
  // fields here
}

record UserAction {
  User user;
  string action;
}

And if you use Avro Maven plugin idl-schemata action, it'll reflect all User changes in both output AVSC schema files.

And when the Java models are created, it'll have all the necessary fields. Still, you need to update all external clients that depend on these models separately.