fix: CI flakes, the slot of a source and the unit test job time (keel#177) #180

Merged

navigator merged 4 commits from fix/ci-flakes-177 into main 2026-10-15 22:06:45 -03:00

Conversation 0 Commits 4 Files changed 15 +127 -12

navigator commented 2026-10-15 18:11:38 -03:00

Owner

Copy link

Fixes part of keel#177. The other part is keel#167: the etcd flakes come from the same stalled peer stream, and the PR for keel#167 fixes them.

Classification of the red checks of 2026-10-14 and 2026-10-15

I read all 21 failed jobs of the Forgejo run history (runs 2 to 88).

Check	Test	Cause	Fix
mesh, tests	TestBlocked::test_a_source_refused_too_often_is_not_answered (runs 28, 55, 70, 88)	Ordering race in the product. The listener counts one connection per source, and the thread freed the count only after it closed the connection. The client has its answer before that, asks again, and the listener closes the new connection unread: UNEXPECTED_EOF_WHILE_READING.	This PR: the handler frees the slot before the answer leaves. The members' channel has the same race (2 per source) and gets the same fix.
tests	test_mesh_bridge::test_forged_requests_cancel_the_invite_through_it (run 88)	Ordering race in the test: it read the code of the listener's thread before that thread returned it.	This PR: the test joins the thread.
tests	killed at 30 min (runs 24, 84)	Time limit. Passing runs took 24 to 29 min. About 10 min went to the 5 mesh netns tests, which run again in "mesh / trixie" and add no coverage (their drivers are other processes).	This PR: they run in the mesh job alone (KEEL_MESH_NETNS). The limit of 30 min stays.
etcd	learner: etcdserver: unhealthy cluster (runs 10, 66)	keel#167.	keel#182 (merged).
etcd-auth	context deadline exceeded, read of 2049 ms (runs 42, 62)	keel#167 class: etcd calls that wait on a stalled peer stream.	keel#182 (merged), then measured again.
vip	f, g: did not add the VIP within 15 s, etcdctl txn: no answer within 10.5 s (runs 26, 68)	keel#167 class.	keel#182 (merged).
postgres	b: b_a_holds_vip false (runs 58, 76)	keel#167 class: the holder lost its renewals while the replica's site was cut.	keel#182 (merged).
postgres	c: c_b_sync_names, expired (runs 8, 14, 24)	Code before keel#172 and keel#174. Not seen after run 24.	None.
vip	test_the_lease_is_the_decided_one (run 55)	The test of keel#174 before the PR changed 10 s to 15 s. Not a flake.	None.
vip-upgrade	dropped_s 5.7 (run 8)	Before keel#174 (release after 10 s). Not seen after.	None.
vip-upgrade	gap of 1.38 s against MAX_BLIP 1.0 (run 74)	One time only. 27 lost pings in a row is not random loss, and the job keeps no ping log, so the cause is not known.	Not changed. MAX_BLIP stays.
tests	wireguard-tools missing (run 2)	Runner host at the cutover. Fixed on 2026-10-14.	None.

Found after this PR opened: mesh / trixie of runs 94 and 102 failed 10 min after the last log line, while pytest -q still ran. git.pop.coop keeps Forgejo's default ZOMBIE_TASK_TIMEOUT of 10 min; runner9's journal says ReportLog error: already_exists: log file has been archived. Fixed in the CI tooling: lxc-trixie.yml prints a line each minute (Keel-Linux/.github#25, merged), and docs/infra/keel-lxc-2.md says so (Keel-Linux/handbook#87, merged).

Changes

• keel/mesh/listener.py, keel/mesh/memberlink.py: the handler calls release() before it sends the answer. The release is done one time only. The slot stays taken while the request is read and decided, so the limits per source and in total do not change.
• tests/test_mesh_channel.py, tests/test_mesh_memberlink.py: new tests make the thread close its connection 0.5 s late. They fail on main with the CI signature, and pass with the fix.
• tests/test_mesh_bridge.py: the test joins the listener's thread.
• 5 mesh netns tests skip in CI unless KEEL_MESH_NETNS is set. .forgejo and .github tests.yml set it in the mesh job.
• keel 0.23.32 (0.23.30 went to keel#181, 0.23.31 to keel#182).

Tests

• The new tests fail on main and pass on this branch.
• The unit tests without the netns suites: 3833 passed, 4 skipped, 478 s on the dev host.

Fixes part of keel#177. The other part is keel#167: the etcd flakes come from the same stalled peer stream, and the PR for keel#167 fixes them. ## Classification of the red checks of 2026-10-14 and 2026-10-15 I read all 21 failed jobs of the Forgejo run history (runs 2 to 88). | Check | Test | Cause | Fix | | --- | --- | --- | --- | | mesh, tests | `TestBlocked::test_a_source_refused_too_often_is_not_answered` (runs 28, 55, 70, 88) | Ordering race in the product. The listener counts one connection per source, and the thread freed the count only after it closed the connection. The client has its answer before that, asks again, and the listener closes the new connection unread: `UNEXPECTED_EOF_WHILE_READING`. | This PR: the handler frees the slot before the answer leaves. The members' channel has the same race (2 per source) and gets the same fix. | | tests | `test_mesh_bridge::test_forged_requests_cancel_the_invite_through_it` (run 88) | Ordering race in the test: it read the code of the listener's thread before that thread returned it. | This PR: the test joins the thread. | | tests | killed at 30 min (runs 24, 84) | Time limit. Passing runs took 24 to 29 min. About 10 min went to the 5 mesh netns tests, which run again in "mesh / trixie" and add no coverage (their drivers are other processes). | This PR: they run in the mesh job alone (`KEEL_MESH_NETNS`). The limit of 30 min stays. | | etcd | learner: `etcdserver: unhealthy cluster` (runs 10, 66) | keel#167. | keel#182 (merged). | | etcd-auth | `context deadline exceeded`, read of 2049 ms (runs 42, 62) | keel#167 class: etcd calls that wait on a stalled peer stream. | keel#182 (merged), then measured again. | | vip | f, g: `did not add the VIP within 15 s`, `etcdctl txn: no answer within 10.5 s` (runs 26, 68) | keel#167 class. | keel#182 (merged). | | postgres | b: `b_a_holds_vip` false (runs 58, 76) | keel#167 class: the holder lost its renewals while the replica's site was cut. | keel#182 (merged). | | postgres | c: `c_b_sync_names`, `expired` (runs 8, 14, 24) | Code before keel#172 and keel#174. Not seen after run 24. | None. | | vip | `test_the_lease_is_the_decided_one` (run 55) | The test of keel#174 before the PR changed 10 s to 15 s. Not a flake. | None. | | vip-upgrade | `dropped_s` 5.7 (run 8) | Before keel#174 (release after 10 s). Not seen after. | None. | | vip-upgrade | gap of 1.38 s against MAX_BLIP 1.0 (run 74) | One time only. 27 lost pings in a row is not random loss, and the job keeps no ping log, so the cause is not known. | Not changed. MAX_BLIP stays. | | tests | `wireguard-tools` missing (run 2) | Runner host at the cutover. Fixed on 2026-10-14. | None. | Found after this PR opened: `mesh / trixie` of runs 94 and 102 failed 10 min after the last log line, while pytest -q still ran. git.pop.coop keeps Forgejo's default ZOMBIE_TASK_TIMEOUT of 10 min; runner9's journal says `ReportLog error: already_exists: log file has been archived`. Fixed in the CI tooling: lxc-trixie.yml prints a line each minute (Keel-Linux/.github#25, merged), and docs/infra/keel-lxc-2.md says so (Keel-Linux/handbook#87, merged). ## Changes - `keel/mesh/listener.py`, `keel/mesh/memberlink.py`: the handler calls `release()` before it sends the answer. The release is done one time only. The slot stays taken while the request is read and decided, so the limits per source and in total do not change. - `tests/test_mesh_channel.py`, `tests/test_mesh_memberlink.py`: new tests make the thread close its connection 0.5 s late. They fail on main with the CI signature, and pass with the fix. - `tests/test_mesh_bridge.py`: the test joins the listener's thread. - 5 mesh netns tests skip in CI unless `KEEL_MESH_NETNS` is set. `.forgejo` and `.github` tests.yml set it in the mesh job. - keel 0.23.32 (0.23.30 went to keel#181, 0.23.31 to keel#182). ## Tests - The new tests fail on main and pass on this branch. - The unit tests without the netns suites: 3833 passed, 4 skipped, 478 s on the dev host.

navigator added 4 commits 2026-10-15 18:11:38 -03:00

test: a source that has its answer asks again at once (keel#177) ... 550446dff4

The invite's listener and the members' channel count one connection
per source. Each thread frees that count only after it closes its
connection. On a loaded machine, the source asks again before that,
and the listener closes the new connection unread. CI showed this as
"the TLS handshake failed (UNEXPECTED_EOF_WHILE_READING)" in
TestBlocked of tests/test_mesh_channel.py.

The new tests make the thread close its connection 0.5 s late. They
fail on main.

fix: the slot of a source is free before its answer leaves (keel#177) ... c275110cda

The handler of the invite's listener and of the members' channel now
frees the slot and the count of its source before it sends the
answer. The source can then ask again at once. The release is done
one time only, so the thread does not free it again when it closes
the connection.

The bridge test now waits for the code of the listener's thread. The
bridge ends when the listener closes its socket, and the thread
returns its code after that.

ci: the mesh netns tests run in the mesh job alone (keel#177) ... 0ab2eeb344

The unit test job "tests / coverage" has a limit of 30 min. Its runs
took 24 to 29 min, and 2 runs of 2026-10-15 hit the limit. About
10 min of each run went to the 5 netns tests of the mesh, which run
again in the job "mesh / trixie". They start their drivers as other
processes, so they add no coverage to the unit test job.

These tests now skip in CI unless KEEL_MESH_NETNS is set, as the etcd
and VIP netns tests do with their own variables. The mesh job sets
it. The unit tests alone take 8 min on a quiet machine.

chore: keel 0.23.30 (keel#177) 2fed410e91

navigator force-pushed fix/ci-flakes-177 from 2fed410e91 to 55aaa18269 2026-10-15 19:18:57 -03:00 Compare

navigator force-pushed fix/ci-flakes-177 from 55aaa18269 to 14f6903f94 2026-10-15 20:07:57 -03:00 Compare

navigator force-pushed fix/ci-flakes-177 from 14f6903f94 to 7efd036afe 2026-10-15 20:48:36 -03:00 Compare

navigator force-pushed fix/ci-flakes-177 from 7efd036afe to 48011e43c2 2026-10-15 21:42:11 -03:00 Compare

navigator merged commit 384afa6bfe into main 2026-10-15 22:06:45 -03:00

navigator referenced this pull request from a commit 2026-10-15 22:06:47 -03:00

Merge pull request 'fix: CI flakes, the slot of a source and the unit test job time (keel#177)' (#180) from fix/ci-flakes-177 into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

accessibility

Barrier affecting people with disabilities

  

bug

Something isn't working

  

documentation

Improvements or additions to documentation

  

duplicate

This issue or pull request already exists

  

enhancement

New feature or request

  

good first issue

Good for newcomers

  

help wanted

Extra attention is needed

  

invalid

This doesn't seem right

  

question

Further information is requested

  

wontfix

This will not be worked on

No labels

accessibility

bug

documentation

duplicate

enhancement

good first issue

help wanted

invalid

question

wontfix

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

Keel-Linux/keel!180

No description provided.