vip: a cut of the site that holds the etcd leader drops the VIP on the holder for 2 to 4 s; once, under netem, etcd had no leader for 111 s #167

New issue

Open

opened 2026-10-15 08:42:25 -03:00 by navigator · 2 comments

navigator commented

2026-10-15 08:42:25 -03:00

Owner

Found on real nodes on 2026-10-15 (keel 0.23.21, etcd of 3 members in 3 sites: 9054 BR1, 9062 BR2, witness 9071 BR3; handbook decision 0057, evidence docs/evidence/2026-10-15-br3-thin-etcd-witness.md).

Short drops

Each cut drops all WireGuard packets of one CT. The VIP holder is not cut.

Cut	Network	etcd leader at the cut	VIP on the holder
witness	base (4 ms and 43 ms)	witness	dropped 6.9 s after the cut, back 2.3 s later
witness	netem, 254 ms and 285 ms, 1.7 to 2 % loss	witness	dropped 7.0 s after the cut, back 2.3 s later
standby 9062	base	9062	kept
standby 9062	netem (2nd run)	9062	dropped 7.0 s after the cut, back 3.9 s later

At base, the new leader was elected 7.4 s after the cut (election timeout 5 s, pre-vote). The holder drops the VIP 10 s after its last renewal that the majority confirmed. Thus the margin of docs/vip.md ("10 s rides out one re-election without a move") does not hold when the leader is lost: the drop comes before the new leader answers. A write in flight waits, and the writes stop for the 2 to 4 s.

No leader for 111 s (netem, 1st run of the standby cut)

The cut of 9062 (the leader) started at 1792063794.43. 9054 and the witness each sent MsgPreVote every 5.4 to 7.8 s for 111 s, and no leader was elected until 1792063913.27, 47 s after the heal:

9054 rejected the witness's pre-vote: "[logterm: 6, index: 128] rejected MsgPreVote from 4d056d928326e022 [logterm: 6, index: 126]". The witness was 2 entries behind.
The witness got none of 9054's messages during the cut. At 1792063915.83 it logged a burst of 9054's MsgAppResp and MsgPreVoteResp "with lower term": they came after the heal.
An overlay ping between 9054 and the witness during a 2nd run of the same cut: 0 % loss, 282 to 290 ms.

The VIP was off every node for the whole time, and the server on 9054 was stopped (keel#168). The 2nd run of the same cut gave a normal election.

Questions

Does keel need a release time longer than one election under loss, or a way to keep the witness from the leader role (etcdctl move-leader in tend)?
Why did the peer stream from 9054 to the witness stall? A stream stalled behind a lost leader is enough to lose the quorum of 2 of 3.

Found on real nodes on 2026-10-15 (keel 0.23.21, etcd of 3 members in 3 sites: 9054 BR1, 9062 BR2, witness 9071 BR3; handbook decision 0057, evidence `docs/evidence/2026-10-15-br3-thin-etcd-witness.md`). ## Short drops Each cut drops all WireGuard packets of one CT. The VIP holder is not cut. | Cut | Network | etcd leader at the cut | VIP on the holder | | --- | --- | --- | --- | | witness | base (4 ms and 43 ms) | witness | dropped 6.9 s after the cut, back 2.3 s later | | witness | netem, 254 ms and 285 ms, 1.7 to 2 % loss | witness | dropped 7.0 s after the cut, back 2.3 s later | | standby 9062 | base | 9062 | kept | | standby 9062 | netem (2nd run) | 9062 | dropped 7.0 s after the cut, back 3.9 s later | At base, the new leader was elected 7.4 s after the cut (election timeout 5 s, pre-vote). The holder drops the VIP 10 s after its last renewal that the majority confirmed. Thus the margin of `docs/vip.md` ("10 s rides out one re-election without a move") does not hold when the leader is lost: the drop comes before the new leader answers. A write in flight waits, and the writes stop for the 2 to 4 s. ## No leader for 111 s (netem, 1st run of the standby cut) The cut of 9062 (the leader) started at 1792063794.43. 9054 and the witness each sent `MsgPreVote` every 5.4 to 7.8 s for 111 s, and no leader was elected until 1792063913.27, 47 s after the heal: - 9054 rejected the witness's pre-vote: "[logterm: 6, index: 128] rejected MsgPreVote from 4d056d928326e022 [logterm: 6, index: 126]". The witness was 2 entries behind. - The witness got none of 9054's messages during the cut. At 1792063915.83 it logged a burst of 9054's `MsgAppResp` and `MsgPreVoteResp` "with lower term": they came after the heal. - An overlay ping between 9054 and the witness during a 2nd run of the same cut: 0 % loss, 282 to 290 ms. The VIP was off every node for the whole time, and the server on 9054 was stopped (keel#168). The 2nd run of the same cut gave a normal election. ## Questions - Does keel need a release time longer than one election under loss, or a way to keep the witness from the leader role (`etcdctl move-leader` in tend)? - Why did the peer stream from 9054 to the witness stall? A stream stalled behind a lost leader is enough to lose the quorum of 2 of 3.

navigator referenced this issue

2026-10-15 09:56:36 -03:00

fix: the VIP rides out an etcd re-election, and the watch starts a stopped holder (keel#167, keel#168) #174

navigator commented

2026-10-15 09:57:08 -03:00

Author

Owner

Analysis, 2026-10-15. PR #174 fixes the short drop. The 111 s with no leader stays open here.

The short drop (fixed in #174)

The kernel ended the address int(10 - age) - 1 s after the carry, so about 8 s after the last renewal that the majority confirmed. The new leader came 7.4 s after the cut at base. The release is now 15 s: the kernel keeps the address up to 14 s after that renewal. etcd expires the lease no sooner than 20 s after its receipt of the renewal, so the margin is 5 s or more.

The 111 s with no leader (not fixed)

What the logs show:

9054 rejected each pre-vote of the witness, because the log of the witness was 2 entries behind (logterm: 6, index: 126 against index: 128). Raft does this by design.
The witness received no message from 9054 during the cut. It logged 9054's MsgAppResp and MsgPreVoteResp as a burst at 1792063915.83, after the heal, "with lower term".
An overlay ping between 9054 and the witness in the 2nd run of the same cut had 0 % loss at 282 to 290 ms.

Thus the path between 9054 and the witness worked, but the raft messages from 9054 to the witness did not arrive until the heal. etcd sends raft messages over a long-lived peer stream (TCP over wg0). The probable cause is that this stream stalled: under 285 ms RTT and loss, a TCP connection that lost several retransmissions waits for an RTO that doubles each time, while new connections and pings work. etcd closes and opens the stream again only when it sees an error.

Why the witness could not win alone: it needs the vote of 9054, and 9054 refuses a candidate with an older log. Why 9054 could not win: its pre-vote did not reach the witness.

Next steps (not done here):

reproduce in the netns harness: cut the etcd leader under the netem profile, and record ss -tin of the etcd peer connections (port 2380 or keel's peer port) on the 2 other members during the cut;
if the stream stalls, set a TCP user timeout or keepalive on the etcd peer connections, or lower net.ipv4.tcp_retries2 for wg0 only;
a longer election timeout does not help a stalled stream.

The 2nd run of the same cut elected a leader normally, so this does not occur on each cut.

Analysis, 2026-10-15. PR #174 fixes the short drop. The 111 s with no leader stays open here. ## The short drop (fixed in #174) The kernel ended the address `int(10 - age) - 1` s after the carry, so about 8 s after the last renewal that the majority confirmed. The new leader came 7.4 s after the cut at base. The release is now 15 s: the kernel keeps the address up to 14 s after that renewal. etcd expires the lease no sooner than 20 s after its receipt of the renewal, so the margin is 5 s or more. ## The 111 s with no leader (not fixed) What the logs show: 1. 9054 rejected each pre-vote of the witness, because the log of the witness was 2 entries behind (`logterm: 6, index: 126` against `index: 128`). Raft does this by design. 2. The witness received no message from 9054 during the cut. It logged 9054's `MsgAppResp` and `MsgPreVoteResp` as a burst at 1792063915.83, after the heal, "with lower term". 3. An overlay ping between 9054 and the witness in the 2nd run of the same cut had 0 % loss at 282 to 290 ms. Thus the path between 9054 and the witness worked, but the raft messages from 9054 to the witness did not arrive until the heal. etcd sends raft messages over a long-lived peer stream (TCP over wg0). The probable cause is that this stream stalled: under 285 ms RTT and loss, a TCP connection that lost several retransmissions waits for an RTO that doubles each time, while new connections and pings work. etcd closes and opens the stream again only when it sees an error. Why the witness could not win alone: it needs the vote of 9054, and 9054 refuses a candidate with an older log. Why 9054 could not win: its pre-vote did not reach the witness. Next steps (not done here): - reproduce in the netns harness: cut the etcd leader under the netem profile, and record `ss -tin` of the etcd peer connections (port 2380 or keel's peer port) on the 2 other members during the cut; - if the stream stalls, set a TCP user timeout or keepalive on the etcd peer connections, or lower `net.ipv4.tcp_retries2` for wg0 only; - a longer election timeout does not help a stalled stream. The 2nd run of the same cut elected a leader normally, so this does not occur on each cut.

navigator referenced this issue from Keel-Linux/handbook

2026-10-15 18:28:27 -03:00

docs: 0050 amendment, a stalled TCP connection ends after 5 retransmissions (keel#167) #84

navigator commented

2026-10-15 22:06:57 -03:00

Author

Owner

The cause of the 111 s is fixed in keel#182 (0.23.31, merged): a peer connection that a cut stalled held raft's messages until its next retransmission, up to 120 s. keel now sets net.ipv4.tcp_retries2 = 5. In the netns harness, 12 leader cuts gave a new leader in 8 to 13 s (before: up to no leader in 40 s). Evidence: evidence/2026-10-15-etcd-stalled-peer-stream.md; handbook#84 amends 0050. This issue stays open until the real-node matrix confirms it, when BR3-THIN is back.

The cause of the 111 s is fixed in keel#182 (0.23.31, merged): a peer connection that a cut stalled held raft's messages until its next retransmission, up to 120 s. keel now sets `net.ipv4.tcp_retries2 = 5`. In the netns harness, 12 leader cuts gave a new leader in 8 to 13 s (before: up to no leader in 40 s). Evidence: `evidence/2026-10-15-etcd-stalled-peer-stream.md`; handbook#84 amends 0050. This issue stays open until the real-node matrix confirms it, when BR3-THIN is back.