vip promote: the etcd claim txn is single-shot, "no answer within 10.5 s" fails the promote on a poor link #135

New issue

Open

opened 2026-10-09 15:19:11 -03:00 by marcos-mendez · 0 comments

marcos-mendez commented

2026-10-09 15:19:11 -03:00

(Migrated from github.com)

Two CI sightings on 2026-10-09, both on the promote path over the lossy
link (250 ms RTT, 2% loss, netem):

PR #132 (mariadb / trixie): test_the_unplanned_write_downtime failed
with downtime_s 24.95 (limit 3.0); promote_took_s 32.55.
PR #134 (vip / trixie, tests/test_vip_netns.py test_f/g): promote
exited 16 with "etcd did not take the claim: etcdctl txn: no answer
within 10.5 s"; the retry then hit "keel-vip.service did not add the
address to wg0 within 15 s".

The claim's etcd txn has one attempt with a 10.5 s timeout. The existing
promote retry (keel#94, PR #128) covers only the etcd read phrase
"etcd did not answer", not the claim txn phrase. On a loaded runner the
txn can outlast 10.5 s and the promote fails even though a bounded retry
would take the claim.

Expected: promote retries the claim a bounded number of times when the
only failure is the txn timeout, as the other single-shot calls do.

Two CI sightings on 2026-10-09, both on the promote path over the lossy link (250 ms RTT, 2% loss, netem): 1. PR #132 (mariadb / trixie): test_the_unplanned_write_downtime failed with downtime_s 24.95 (limit 3.0); promote_took_s 32.55. 2. PR #134 (vip / trixie, tests/test_vip_netns.py test_f/g): promote exited 16 with "etcd did not take the claim: etcdctl txn: no answer within 10.5 s"; the retry then hit "keel-vip.service did not add the address to wg0 within 15 s". The claim's etcd txn has one attempt with a 10.5 s timeout. The existing promote retry (keel#94, PR #128) covers only the etcd read phrase "etcd did not answer", not the claim txn phrase. On a loaded runner the txn can outlast 10.5 s and the promote fails even though a bounded retry would take the claim. Expected: promote retries the claim a bounded number of times when the only failure is the txn timeout, as the other single-shot calls do.