fix: keel vip promote --old-primary-gone waits on the old primary 1.5 s at most (keel#108) #121

Merged

marcos-mendez merged 1 commit from fix/promote-old-primary-gone-bounded into main

2026-10-09 03:48:19 -03:00

marcos-mendez commented

2026-10-09 01:50:15 -03:00

(Migrated from github.com)

Fixes #108 (finding B7 of Keel-Linux/handbook#60, again in the 0.23.7 rollout, Keel-Linux/handbook#62).

Problem

keel vip promote --old-primary-gone took 30.5 s with the old primary dead: three serial waits for the members' channel's 10 s timeout (the epoch, the release, the announcement).

Change (after the safety review)

With etcd formed: the old primary's lease fences it. Each ask of it (the release, the announcement) runs in a thread of its own and waits at most 1.5 s (keel/mesh/vipnode.py, bounded, GONE_BOUND); the other peers are not held up. The promote still waits for the lease to go and never revokes it (decision 0049). The ask goes on after the bound, in a thread that is not a daemon, and the promote waits for it before it returns (vipnode.finished), so a slow live old primary still gets the release and the claim.
Without etcd: nothing fences a live old primary (no lease, no lifetime on its address), so nothing is bounded, as before. docs/vip.md, the help of keel vip promote and keel database promote, and the docstring of commands.promote_vip say that, without etcd, an old primary that is alive but cannot be reached stays writable until it learns the newer claim.
debian/changelog 0.23.8.

Tests

Without etcd, a slow old primary (0.6 s, bound 0.3 s) still releases before the new holder takes the VIP.
With etcd, a hung old primary costs the bound and the late asks are joined; an old primary that answers after the bound still drops the VIP.
tests/mariadb_netns.py: the crash case writes at the VIP from C every 0.1 s (after the replica is connected and semi-synchronous) and reports the write downtime, its parts and the acknowledged rows lost.

Measured (`mariadb / trixie`, etcd, 250 ms ±25 ms, 2 % loss)

Each ask of the dead old primary: 1.5 s (no answer within 1.5 s, never timed out). With etcd the claim waits for the old primary's lease (TTL 20 s, 0049); #122 removes the lag after the VIP lands. 0 acknowledged rows lost.

CI record

Green at 44a4c68. vip / trixie failed once at g_promote_b (etcdctl get: context deadline exceeded, the same failure as main's run 37833124848); I reran that job once (run 37890526886): green. Earlier head 94bae41: mesh / trixie failed once in test_mesh_channel (TLS EOF), rerun once: green.

Fixes #108 (finding B7 of Keel-Linux/handbook#60, again in the 0.23.7 rollout, Keel-Linux/handbook#62). ## Problem `keel vip promote --old-primary-gone` took 30.5 s with the old primary dead: three serial waits for the members' channel's 10 s timeout (the epoch, the release, the announcement). ## Change (after the safety review) - **With etcd formed**: the old primary's lease fences it. Each ask of it (the release, the announcement) runs in a thread of its own and waits at most 1.5 s (`keel/mesh/vipnode.py`, `bounded`, `GONE_BOUND`); the other peers are not held up. The promote still waits for the lease to go and never revokes it (decision 0049). The ask goes on after the bound, in a thread that is not a daemon, and the promote waits for it before it returns (`vipnode.finished`), so a slow live old primary still gets the release and the claim. - **Without etcd**: nothing fences a live old primary (no lease, no lifetime on its address), so nothing is bounded, as before. `docs/vip.md`, the help of `keel vip promote` and `keel database promote`, and the docstring of `commands.promote_vip` say that, without etcd, an old primary that is alive but cannot be reached stays writable until it learns the newer claim. - `debian/changelog` 0.23.8. ## Tests - Without etcd, a slow old primary (0.6 s, bound 0.3 s) still releases before the new holder takes the VIP. - With etcd, a hung old primary costs the bound and the late asks are joined; an old primary that answers after the bound still drops the VIP. - `tests/mariadb_netns.py`: the crash case writes at the VIP from C every 0.1 s (after the replica is connected and semi-synchronous) and reports the write downtime, its parts and the acknowledged rows lost. ## Measured (`mariadb / trixie`, etcd, 250 ms ±25 ms, 2 % loss) Each ask of the dead old primary: 1.5 s (`no answer within 1.5 s`, never `timed out`). With etcd the claim waits for the old primary's lease (TTL 20 s, 0049); #122 removes the lag after the VIP lands. 0 acknowledged rows lost. ## CI record Green at 44a4c68. `vip / trixie` failed once at `g_promote_b` (`etcdctl get: context deadline exceeded`, the same failure as main's run 37833124848); I reran that job once (run 37890526886): green. Earlier head 94bae41: `mesh / trixie` failed once in `test_mesh_channel` (TLS EOF), rerun once: green.