ci: flaky tests cost about 1 h per PR (mesh channel TLS EOF, vip carry wait, etcd deadline) #177

New issue

Closed

opened 2026-10-15 14:32:15 -03:00 by navigator · 1 comment

navigator commented

2026-10-15 14:32:15 -03:00

Owner

The CI checks of 2026-10-15 failed often on known flakes. Each failure passed on a re-run, and none came from the change under test.

Job	Test	Signature	PRs
mesh	`tests/test_mesh_channel.py::TestBlocked::test_a_source_refused_too_often_is_not_answered`	`Forged: [::1]:N: the TLS handshake failed ([SSL: UNEXPECTED_EOF_WHILE_READING] ...)`	#172, #174, #175
vip	`test_g_a_member_outside_the_pair_writing_and_revoking`	`keel-vip.service did not add the VIP to wg0 within 15 s` (the promote's carry wait, under netem)	#171
etcd-auth	`test_the_move_leaves_the_vip_s_holder_alone`	`etcdctl get: context deadline exceeded` (keel#94 class)	#173
etcd-auth	`test_etcdctl_meets_the_vip_s_renewal`	read max 2049.5 ms against the call timeout of 2000 ms	#175
etcd	`test_a_learner_added_and_removed_with_keel_s_client`	`etcdserver: unhealthy cluster` (keel#94 class)	#175
vip-upgrade	`test_a_the_holder_restarted_as_an_upgrade_does`	`dropped_s` not None after try-restart	#170

Forgejo has no API to re-run one job. A re-run is a new commit date and a force-push, and every job runs again. With 6 runners a full run takes 45 to 60 min, so each flake costs about 1 h.

Expected: each of these tests passes on the design case, or it is fixed or bounded so that it does not fail a PR on a loaded runner.

The CI checks of 2026-10-15 failed often on known flakes. Each failure passed on a re-run, and none came from the change under test. | Job | Test | Signature | PRs | | --- | --- | --- | --- | | mesh | `tests/test_mesh_channel.py::TestBlocked::test_a_source_refused_too_often_is_not_answered` | `Forged: [::1]:N: the TLS handshake failed ([SSL: UNEXPECTED_EOF_WHILE_READING] ...)` | #172, #174, #175 | | vip | `test_g_a_member_outside_the_pair_writing_and_revoking` | `keel-vip.service did not add the VIP to wg0 within 15 s` (the promote's carry wait, under netem) | #171 | | etcd-auth | `test_the_move_leaves_the_vip_s_holder_alone` | `etcdctl get: context deadline exceeded` (keel#94 class) | #173 | | etcd-auth | `test_etcdctl_meets_the_vip_s_renewal` | read max 2049.5 ms against the call timeout of 2000 ms | #175 | | etcd | `test_a_learner_added_and_removed_with_keel_s_client` | `etcdserver: unhealthy cluster` (keel#94 class) | #175 | | vip-upgrade | `test_a_the_holder_restarted_as_an_upgrade_does` | `dropped_s` not None after try-restart | #170 | Forgejo has no API to re-run one job. A re-run is a new commit date and a force-push, and every job runs again. With 6 runners a full run takes 45 to 60 min, so each flake costs about 1 h. Expected: each of these tests passes on the design case, or it is fixed or bounded so that it does not fail a PR on a loaded runner.

navigator referenced this issue from Keel-Linux/.github

2026-10-15 20:38:24 -03:00

ci: lxc-trixie prints a line each minute while the commands run (keel#177) #25

navigator referenced this issue from Keel-Linux/handbook

2026-10-15 20:48:59 -03:00

docs: infra, a job silent for 10 min fails on git.pop.coop (keel#177) #87

navigator commented

2026-10-15 22:06:57 -03:00

Author

Owner

Fixed by keel#180 (0.23.32), keel#182 (0.23.31), Keel-Linux/.github#25 and handbook#87, all merged. The classification of each red check is in the description of keel#180. One item stays unknown: the vip-upgrade gap of 1.38 s of run 74 (one time, no ping log kept). If it comes again, open a new issue with the ping log.

navigator closed this issue

2026-10-15 22:06:57 -03:00