How to ensure consistent reading in distributed system?

Question

In a distributed system, if only half of the nodes are successfully written, the subsequent nodes that read the unwritten data will be inconsistent. How to avoid this situation?

client write --> Node1  v
             --> Node2  v
client read  --> Node3  x(The latest data was not read)

My plan:

• Compare the data version with other nodes when reading data
• If the current node version is found to be lower, it will be routed to other nodes to read data.

Answer 1

I am going to ignore tags [mongo and elastic] :)

What you are planning to do is called Dynamo style replication. That system is eventually consistent by design. (I read a while ago that it could get strongly consistent with some effort, but I don't remember if that paper was correct.)

Back to dynamo and quorum: with three nodes you want to have at least 2 nodes to save writes to assume the write has succeeded. Important point is that you need two nodes to report back to customer the success, but the data is still should to be sent to three nodes.

Let's assume that data is written to two nodes, third failed, but camed back online later. To read the data, you have to read it from any two nodes as well. You will sent read requests to all three, but only two is needed to report back to the customer. This will give you quorum: 2+2>3. This guarantees that there is an intersection in between writes and reads.

This will work ok when the network is good and nodes are healthy. But you will run into major challengers, lost updates and conflict resolution to name a few. But in either way, the system will not be strongly consistent based on design itself.

Let me describe another interesting issue to illustrate weak consistency:

• node 1 gets the write
• the rest of process fails; node 1 has new data, but node 2 and 3 don't
• now, when you read, under the quorum condition, you may or may not see the value from node 1 - since you are picking any two nodes for a read, node 1 may not be in that set.

Long story short, dynamo is not good for strong consistency, and we get to the Raft part of the solution.

Raft will get you what you need. A consistent system. There is a catch to watch for. Most examples are focused on writing - raft maintains a log of messages and consensus is used to agree on the order (and content) of these messages.

But when you do a read, you can't just go to a node, or any two nodes, or three and read the value. You will have to do read via Raft as well, by attaching a read operation to raft's log. This is called linearizable read.

I'll stop here, as this is pretty complicated topic (but not an impossible one to learn).

Hope this gave you enough ideas to explore.