fix: a stalled etcd peer connection ends after 5 retransmissions (keel#167) #182

Merged

navigator merged 2 commits from fix/etcd-stalled-peer-stream-167 into main 2026-10-15 21:31:05 -03:00

Conversation 0 Commits 2 Files changed 13 +269 -8

navigator commented 2026-10-15 18:28:46 -03:00

Owner

Copy link

Fixes keel#167 (the 111 s with no leader). Also fixes the etcd class of keel#177: the learner's unhealthy cluster, the etcd-auth deadlines, the vip f/g promotes and the postgres b drop all came after a cut, while a peer stream still waited for a retransmission.

Cause

Reproduced in the netns harness at 250 ms ±25 ms with 2% loss: 3 members, 12 cuts of the leader for 40 s. A peer connection that lost its segments in a cut doubles its retransmission timeout, up to the kernel's 120 s. etcd goes on writing raft's messages to it and sees no error (the send queue grew from 4347 to 8343 bytes, backoff 7, RTO 58 s, 80 s after the heal). A survivor's votes wait in that connection, so the election waits for the next retransmission. One cut had no leader for the whole 40 s. This matches the real nodes: the witness got none of 9054's messages, then a burst of old ones after the heal.

The etcd tuning that keel renders cannot end such a connection:

• heartbeat and election timeout: a longer election does not unblock a stalled connection;
• --raft-read-timeout and --raft-write-timeout (5 s): the write goes into the kernel's buffer and does not fail; the reader side had already closed, and its FIN was lost in the cut;
• gRPC keepalive: for client connections only, and etcdctl opens a new process for each call.

Fix

keel writes /etc/sysctl.d/90-keel-etcd.conf with net.ipv4.tcp_retries2 = 5, and apply sets it live with sysctl -w (no restart: the kernel reads it at each retransmission). The kernel ends a stalled connection after about 14 s, and etcd dials again. The value is per network namespace, for IPv4 and IPv6. Its effect on the other TCP users of the node and the false-kill risk (6 losses of one segment in a row at 2%: once in 10^10) are in the docstring of keel/mesh/etcdconf.py, docs/mesh.md and the handbook amendment of 0050 (Keel-Linux/handbook#84).

Test (written first)

tests/etcd_netns.py samples each second the TCP backoff of every etcd peer connection. tests/test_etcd_netns.py now fails when:

• a peer connection reaches a backoff above 5;
• the follower rejoins 20 s or more after the heal (CI before: 19.1 to 51.2 s in all 33 runs);
• the leader cut just after that rejoin has no new leader within 30 s (CI before: 48 to 51 s in 8 of 33 runs; the bound was 60 s).

The netns harnesses of etcd and of the VIP set keel's sysctl file in each namespace, as a node does.

Commit	etcd netns test on the dev host	rejoined_s	new_leader_s	highest backoff
test only	2 failed, 6 passed	20.0	9.0	7 to 8
test and fix	8 passed	6.0	12.0	4

12 leader cuts, without and with the setting: new leader 9 to 28 s and one cut with none in 40 s, against 8 to 13 s; members back 24 to 61 s after the heal, against 8 to 16 s. Evidence: evidence/2026-10-15-etcd-stalled-peer-stream.md.

Unit tests: tests/test_mesh_etcdconf.py, tests/test_system_etcd.py (the file, the live set, nothing under --root). The unit suite without the netns suites: 3833 passed, 4 skipped.

keel 0.23.31. Not released: releases wait for the real-node matrix (BR3-THIN is offline).

Fixes keel#167 (the 111 s with no leader). Also fixes the etcd class of keel#177: the learner's `unhealthy cluster`, the etcd-auth deadlines, the vip f/g promotes and the postgres b drop all came after a cut, while a peer stream still waited for a retransmission. ## Cause Reproduced in the netns harness at 250 ms ±25 ms with 2% loss: 3 members, 12 cuts of the leader for 40 s. A peer connection that lost its segments in a cut doubles its retransmission timeout, up to the kernel's 120 s. etcd goes on writing raft's messages to it and sees no error (the send queue grew from 4347 to 8343 bytes, backoff 7, RTO 58 s, 80 s after the heal). A survivor's votes wait in that connection, so the election waits for the next retransmission. One cut had no leader for the whole 40 s. This matches the real nodes: the witness got none of 9054's messages, then a burst of old ones after the heal. The etcd tuning that keel renders cannot end such a connection: - heartbeat and election timeout: a longer election does not unblock a stalled connection; - `--raft-read-timeout` and `--raft-write-timeout` (5 s): the write goes into the kernel's buffer and does not fail; the reader side had already closed, and its FIN was lost in the cut; - gRPC keepalive: for client connections only, and etcdctl opens a new process for each call. ## Fix keel writes `/etc/sysctl.d/90-keel-etcd.conf` with `net.ipv4.tcp_retries2 = 5`, and apply sets it live with `sysctl -w` (no restart: the kernel reads it at each retransmission). The kernel ends a stalled connection after about 14 s, and etcd dials again. The value is per network namespace, for IPv4 and IPv6. Its effect on the other TCP users of the node and the false-kill risk (6 losses of one segment in a row at 2%: once in 10^10) are in the docstring of `keel/mesh/etcdconf.py`, `docs/mesh.md` and the handbook amendment of 0050 (Keel-Linux/handbook#84). ## Test (written first) `tests/etcd_netns.py` samples each second the TCP backoff of every etcd peer connection. `tests/test_etcd_netns.py` now fails when: - a peer connection reaches a backoff above 5; - the follower rejoins 20 s or more after the heal (CI before: 19.1 to 51.2 s in all 33 runs); - the leader cut just after that rejoin has no new leader within 30 s (CI before: 48 to 51 s in 8 of 33 runs; the bound was 60 s). The netns harnesses of etcd and of the VIP set keel's sysctl file in each namespace, as a node does. | Commit | etcd netns test on the dev host | rejoined_s | new_leader_s | highest backoff | | --- | --- | --- | --- | --- | | test only | 2 failed, 6 passed | 20.0 | 9.0 | 7 to 8 | | test and fix | 8 passed | 6.0 | 12.0 | 4 | 12 leader cuts, without and with the setting: new leader 9 to 28 s and one cut with none in 40 s, against 8 to 13 s; members back 24 to 61 s after the heal, against 8 to 16 s. Evidence: `evidence/2026-10-15-etcd-stalled-peer-stream.md`. Unit tests: `tests/test_mesh_etcdconf.py`, `tests/test_system_etcd.py` (the file, the live set, nothing under `--root`). The unit suite without the netns suites: 3833 passed, 4 skipped. keel 0.23.31. Not released: releases wait for the real-node matrix (BR3-THIN is offline).

navigator added 2 commits 2026-10-15 18:28:46 -03:00

test: no etcd peer connection waits in a deep backoff (keel#167) ... 5d0e32cb7b

The netns harness of etcd now samples each second the retransmission
backoff of every etcd peer connection, with ss. The test fails when a
connection reaches a backoff above 5. On main it reaches 6 to 8 in the
cuts of 45 s and 120 s. Such a connection holds raft's messages until
its next retransmission, up to 120 s, also after the heal.

Two bounds of the test now catch the stall itself:

- the follower rejoins within 20 s of the heal (19 to 51 s in CI);
- the leader cut just after that rejoin has a new leader within 30 s
  (48 to 51 s in 8 of 33 CI runs).

fix: a stalled etcd peer connection ends after 5 retransmissions (keel#167) ... 30ba87ac75

On real nodes under netem, etcd had no leader for 111 s after a cut of
its leader. The netns harness shows the cause. A peer connection that
loses its segments in a cut doubles its retransmission timeout up to
the kernel's 120 s. etcd goes on writing raft's messages to it and
sees no error, so the messages wait for the next retransmission, also
after the heal. A survivor's votes then do not reach the other
survivor. etcd has no setting for this: its heartbeat, election
timeout and raft read and write timeouts do not end such a connection,
and its gRPC keepalive is for clients only.

keel now writes /etc/sysctl.d/90-keel-etcd.conf with
net.ipv4.tcp_retries2 = 5, and apply sets it live with sysctl -w. The
kernel ends such a connection after about 14 s of retransmissions, and
etcd dials again. The value holds for each TCP connection of the node,
IPv6 included. At 2% loss, 6 losses of one segment in a row come once
in 10^10 segments, so a working connection is not ended.

The netns harnesses of etcd and of the VIP set keel's sysctl file in
each namespace. In 12 leader cuts of 40 s, the new leader came 8 to
13 s after the cut, and all members were back 8 to 16 s after the heal.
Without the setting: 9 to 28 s, one cut with no leader in 40 s, and
24 to 61 s. The evidence is evidence/2026-10-15-etcd-stalled-peer-stream.md.

keel 0.23.31.

navigator force-pushed fix/etcd-stalled-peer-stream-167 from 30ba87ac75 to 8b65d1dc45 2026-10-15 20:58:44 -03:00 Compare

navigator merged commit b1caa07d2a into main 2026-10-15 21:31:05 -03:00

navigator referenced this pull request from a commit 2026-10-15 21:31:07 -03:00

Merge pull request 'fix: a stalled etcd peer connection ends after 5 retransmissions (keel#167)' (#182) from fix/etcd-stalled-peer-stream-167 into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

accessibility

Barrier affecting people with disabilities

  

bug

Something isn't working

  

documentation

Improvements or additions to documentation

  

duplicate

This issue or pull request already exists

  

enhancement

New feature or request

  

good first issue

Good for newcomers

  

help wanted

Extra attention is needed

  

invalid

This doesn't seem right

  

question

Further information is requested

  

wontfix

This will not be worked on

No labels

accessibility

bug

documentation

duplicate

enhancement

good first issue

help wanted

invalid

question

wontfix

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

Keel-Linux/keel!182

No description provided.