Prevent data loss while upgrading Kafka with a single broker

Question

I have a Kafka server which runs on a single node. There is only 1 node because it's a test server. But even for a test server, I need to be sure that no data loss will occur while upgrade is in process.

I upgrade Kafka as:

1. Stop Kafka, Zookeeper, Kafka Connect and Schema Registry.
2. Upgrade all the components.
3. Start upgraded services.

Data loss may occur in the first step, where kafka is not running. I guess you can do a rolling update (?) with multiple brokers to prevent data loss but in my case it is not possible. How can I do something similar with a single broker? Is it possible? If not, what is the best approach for upgrading?

Answer 1

I have to say, obviously, you are always vulnerable to data losses if you are using only one node.

If you can't have more nodes you have the only choice:

1. Stop producing;
2. Stop consuming;
3. Enable parameter controlled.shutdown.enable - this will ensure that your broker saved offset in case of a shutdown.
   I guess the first 2 steps are quite tricky.

Unfortunately, there is not much to play with - Kafka was not designed to be fault-tolerant with only one node.

Answer 2

The process of a rolling upgrade is still the same for a single broker.

Existing data during the upgrade shouldn't be lost.

Obviously, if producers are still running, all their requests will be denied while the broker is down, thus why you not only need multiple brokers to prevent data-loss, but a balanced cluster (with unclean leader election disabled) where your restart cycles don't completely take a set of topics offline.