Can anyone explain this etcd behavior?

Question

Let me ask a question on a strange etcd behavior.

environment:

-bash-4.2$ etcd --version
etcd Version: 3.2.28
Git SHA: 2d861f3
Go Version: go1.10.3
Go OS/Arch: linux/amd64

-bash-4.2$ etcdctl cluster-health
member ef05587d2e4769f is healthy: got healthy result from http://10.7.211.15:2379
member 6066465b170c501d is healthy: got healthy result from http://10.7.211.13:2379
member 7132cc73aebdbcb8 is healthy: got healthy result from http://10.10.51.17:2379
member 7eb23e55f039af25 is healthy: got healthy result from http://10.7.211.14:2379
member c60f0881d3524793 is healthy: got healthy result from http://10.7.211.12:2379
cluster is healthy

what happend:

• member ef05587d2e4769f on host 10.7.211.15 had been the leader
• host 10.7.211.15 had a defect resulting in ethernet flipping
• member ef05587d2e4769f also flipped between available and unavailable therefore

1. the cluster voted a new leader but changed the leader back to member ef05587d2e4769f every time it was available again
2. because member ef05587d2e4769f soon went unavailable again the cluster voted a new leader...

• this loop repeated for hours
• the whole cluster got kind of blocked because of permanent leader election

...
Feb 03 05:58:34 szhm58466 etcd[1027]: raft.node: 6066465b170c501d changed leader from 7132cc73aebdbcb8 to 7eb23e55f039af25 at term 3349
Feb 03 05:58:43 szhm58466 etcd[1027]: raft.node: 6066465b170c501d changed leader from 6066465b170c501d to ef05587d2e4769f at term 3352
Feb 03 06:10:34 szhm58466 etcd[1027]: raft.node: 6066465b170c501d changed leader from 6066465b170c501d to ef05587d2e4769f at term 3371
Feb 03 06:17:42 szhm58466 etcd[1027]: raft.node: 6066465b170c501d changed leader from 6066465b170c501d to ef05587d2e4769f at term 3373
Feb 03 06:23:47 szhm58466 etcd[1027]: raft.node: 6066465b170c501d changed leader from 6066465b170c501d to ef05587d2e4769f at term 3375
Feb 03 06:30:58 szhm58466 etcd[1027]: raft.node: 6066465b170c501d changed leader from 6066465b170c501d to ef05587d2e4769f at term 3379
Feb 03 06:31:01 szhm58466 etcd[1027]: raft.node: 6066465b170c501d changed leader from ef05587d2e4769f to c60f0881d3524793 at term 3379
Feb 03 06:36:46 szhm58466 etcd[1027]: raft.node: 6066465b170c501d changed leader from 6066465b170c501d to ef05587d2e4769f at term 3387
Feb 03 06:37:24 szhm58466 etcd[1027]: raft.node: 6066465b170c501d changed leader from 6066465b170c501d to ef05587d2e4769f at term 3389
Feb 03 06:37:26 szhm58466 etcd[1027]: raft.node: 6066465b170c501d changed leader from ef05587d2e4769f to c60f0881d3524793 at term 3389
Feb 03 07:11:04 szhm58466 etcd[1027]: raft.node: 6066465b170c501d changed leader from 6066465b170c501d to ef05587d2e4769f at term 3400
...

QUESTIONS:

1. why did the cluster change back to the old leader every time the old leader came back again?
2. can this be avoided with some kind of parameter? that means don't change the leader back again anymore once you have changed?

Thanks in advance, Markus

Answer 1

Read there (https://kubernetes.io/blog/2019/08/30/announcing-etcd-3-4/) that the etcd raft voting process has problems with flaky members by design and this should be fixed by the "--pre-vote" feature in etcd v3.4. Correct?

Thanks, Markus

Answer 2

1?why did the cluster change back to the old leader every time the old leader came back again? Answer:According to the definition of the draft algorithm, an election occurs when a membership change occurs. the leader vote process based on network performance, maybe the old leader host is fast network.