mesh: keel mesh etcd form exits 24 with "not reached" for a member that took the cluster #162

New issue

Open

opened 2026-10-15 08:00:15 -03:00 by navigator · 0 comments

navigator commented

2026-10-15 08:00:15 -03:00

Owner

Found on real nodes on 2026-10-15 (keel 0.23.21, handbook decision 0057).

keel mesh etcd form on the root's holder (9054), for a cluster of 3: 9054, 9062 and the witness 9071.

etcd: a cluster of 3 formed from this node; keel mesh status shows its leader once a majority runs
etcd: not reached: fd1a:bb4c:b407::fc0a ([fd1a:bb4c:b407::fc0a]:51821 did not answer: timed out); run keel mesh etcd form again once they answer
rc=24   (38.0 s)

The member 9062 took the cluster. Its etcd started at 1792061627.99, 5 s after the form started. The start waited until a majority ran (keel mesh etcd gate started). That came only at 1792061658.20, after the witness started at 1792061656.11. Thus the answer to the holder came after the timeout of the members' channel. The holder then sent the cluster to the witness, and started its own etcd at 1792061659.07. 10 s later, keel mesh status showed 3 voters, all healthy.

The first member of a new cluster cannot answer before a majority runs, and the form sends the cluster to the members one at a time. Thus the form reports the first member as "not reached", and it exits 24. An operator reads it as a failure.

Expected: the member answers when it took the record and asked etcd to start, before the gate. Or the form sends to all members at the same time, and checks them with member list after its own start.

Found on real nodes on 2026-10-15 (keel 0.23.21, handbook decision 0057). `keel mesh etcd form` on the root's holder (9054), for a cluster of 3: 9054, 9062 and the witness 9071. ``` etcd: a cluster of 3 formed from this node; keel mesh status shows its leader once a majority runs etcd: not reached: fd1a:bb4c:b407::fc0a ([fd1a:bb4c:b407::fc0a]:51821 did not answer: timed out); run keel mesh etcd form again once they answer rc=24 (38.0 s) ``` The member 9062 took the cluster. Its etcd started at 1792061627.99, 5 s after the form started. The start waited until a majority ran (`keel mesh etcd gate started`). That came only at 1792061658.20, after the witness started at 1792061656.11. Thus the answer to the holder came after the timeout of the members' channel. The holder then sent the cluster to the witness, and started its own etcd at 1792061659.07. 10 s later, `keel mesh status` showed 3 voters, all healthy. The first member of a new cluster cannot answer before a majority runs, and the form sends the cluster to the members one at a time. Thus the form reports the first member as "not reached", and it exits 24. An operator reads it as a failure. Expected: the member answers when it took the record and asked etcd to start, before the gate. Or the form sends to all members at the same time, and checks them with `member list` after its own start.