Custom serialization in Kafka using CSharp

Question

I want to write a custom serializer for kafka in CSharp. I've searched a lot and I couldn't find a good reference on how to write custom serializer for kafka in a dotnet language.

Any thoughts on how to write one with dotnet?

Answer 1

I just find the answer to this question. The library to use for kafka in dotnet is provided by confluent.

Kafka .NET Client

There should be a serialization class implementing the interface :

Confluent.Kafka.ISerializer<T>

Normally we should create the producer via ProducerBuilder class :

Confluent.Kafka.ProducerBuilder<TKey, TValue>

There is a method to this class to set serializers for both key and value. It is a little different than the original documentation for custom serializer in Kafka which instructing to set the name of the custom serializer class in the producer configuration.

Following is the code to set the value serializer:

var config = new ProducerConfig
{
    BootstrapServers = "localhost:9092",
    ClientId = "thiPC"
};

var producerBuilder = new ProducerBuilder<Null, Customer>(config);
producerBuilder.SetValueSerializer(new CustomerSerializer());       // <------ The Magic Code
var producer = producerBuilder.Build();

The sample code for the Serialization class is like this;

public class CustomerSerializer : Confluent.Kafka.ISerializer<Customer>
{
    public byte[] Serialize(Customer data, SerializationContext context)
    { ..........}
}

There is nothing important for the Customer class in my example it is a normal class just to hold the customer properties.

In serialization class, you should return a byte[] in the Serialize method which is obvious!

I hope this would be useful for folks implementing Kafka with dotnet.