fix: an old primary stays read only until it proves it holds the VIP (keel#104) #111

Merged

marcos-mendez merged 1 commit from fix/old-primary-read-only-until-proven into main

2026-10-08 22:44:18 -03:00

marcos-mendez commented

2026-10-08 21:50:53 -03:00

(Migrated from github.com)

Fixes #104 (bug B2 of the real-node MariaDB pair test, Keel-Linux/handbook#60).

Problem

After a crash and a boot, the old primary read its own claim of the VIP from /var/lib/keel/vip and keel database follow set read_only OFF. The other node held the VIP at a newer epoch. The old primary was writable for 31 s, until keel vip check learned the newer epoch. A write in that window makes an errant GTID.

Change

keel/system/dbfollow.py: on a node whose newest claim is its own, follow lifts read_only only when the VIP is on wg0. keel adds the VIP to wg0 only after a proof: keel vip promote after its new claim; keel vip check when a peer answered and no peer knows a newer claim; with etcd, the controller after a lease renewal that the majority confirmed (the kernel valid_lft ends it). A boot removes the address. Without the proof, follow sets read_only ON and the root lock, and writes no primary role file. Follow waits up to 15 s (PROOF_WAIT) for the address, because a promote records its claim before it adds the VIP.
keel/mesh/vipnode.py (carry_held): when keel adds the VIP to wg0, it writes the VIP's state again, so keel-database-follow.path runs follow again. keel database watch also runs follow again within 30 s.
No claim on a paired node (the VIP state file removed, or before the first promote): not proven either. A declared primary stays read only and locked, with no primary role file.
keel/system/dbwatch.py (needs_follow): follows a server again when it takes writes with no claim, or while the VIP is not on wg0 (its valid_lft ran out).
keel/system/dbfollow.py (follow): one run at a time, under /var/lib/keel/database/follow.lock, so the unit and keel database watch never interleave between the proof and SET read_only OFF.
Without etcd the address has no lifetime, so the proof is weaker: #113 records it, and docs/replication.md says so.
docs/replication.md: the follow table and the rule.
debian/changelog, pyproject.toml, keel/__init__.py: 0.23.4.

Tests

tests/test_system_dbfollow.py, TestTheOldPrimaryAtBoot: crash, the other node promoted at a newer epoch, the old node boots, stays read only, then follows as a replica (cloud simple, as on the real mesh); a server found writable without a proof is made read only; the holder proven again after its boot takes writes; no peer answers; the proof comes while follow waits; etcd does not answer at boot. The old code fails five of these with SET GLOBAL read_only = OFF.
TestNoClaimAndNoAddress: the state file removed on a primary with a pair record, follow keeps it read only; watch follows a writable primary whose VIP left wg0; follow takes its lock.
tests/test_mesh_vipmove.py: the state is written again when keel vip check adds the VIP, and not when it is there.
tests/mariadb_netns.py: A now takes the VIP (and follow runs) before the application's data is written, because a paired node with no claim is read only.
tests/mariadb_netns.py (mariadb / trixie): the primary's server killed, its VIP helper stopped, cut off; keel database promote --old-primary-gone on the replica; the old primary boots with its old claim: @@read_only is 1 at boot, after follow, and in every 0.5 s sample until it replicates from the new primary over TLS by GTID.

Test plan

tests / coverage
mariadb / trixie (the new crash case)
vip, vip-upgrade, etcd, etcd-auth, mesh / trixie
package / changelog

CI record

First run (617bfbb's first version): mariadb / trixie failed in the new case only: the replica's I/O thread was still Connecting when the status was read (read only held in every sample, replica after 6.4 s). The case now waits for the I/O thread to stream, as case (a) does (amended). vip / trixie failed at g_promote_b (etcdctl get: context deadline exceeded), the same failure as main's run 37833124848 and run 37843439840. No job was rerun: the amended push ran all checks again, all green.

Security review: H1, M2 and L1 are fixed in the amended commit; M1 is #113.

Head 7cbfab1 (after the security review): every check green but mesh / trixie. Attempt 1 timed out in test_a_spent_invite_is_refused (the inviter process). I reran the job once (run 37869774880, attempt 2); it failed in another test, test_c_joins_through_the_fallback, the same failure as main's run 37833124848. This PR changes nothing in keel.mesh's join or invite code. #112 holds this commit plus its own, and its mesh / trixie passed at 132a13a. I did not rerun a second time.

Fixes #104 (bug B2 of the real-node MariaDB pair test, Keel-Linux/handbook#60). ## Problem After a crash and a boot, the old primary read its own claim of the VIP from `/var/lib/keel/vip` and `keel database follow` set `read_only OFF`. The other node held the VIP at a newer epoch. The old primary was writable for 31 s, until `keel vip check` learned the newer epoch. A write in that window makes an errant GTID. ## Change - `keel/system/dbfollow.py`: on a node whose newest claim is its own, follow lifts `read_only` only when the VIP is on `wg0`. keel adds the VIP to `wg0` only after a proof: `keel vip promote` after its new claim; `keel vip check` when a peer answered and no peer knows a newer claim; with etcd, the controller after a lease renewal that the majority confirmed (the kernel `valid_lft` ends it). A boot removes the address. Without the proof, follow sets `read_only ON` and the root lock, and writes no primary role file. Follow waits up to 15 s (`PROOF_WAIT`) for the address, because a promote records its claim before it adds the VIP. - `keel/mesh/vipnode.py` (`carry_held`): when keel adds the VIP to `wg0`, it writes the VIP's state again, so `keel-database-follow.path` runs follow again. `keel database watch` also runs follow again within 30 s. - No claim on a paired node (the VIP state file removed, or before the first promote): not proven either. A declared primary stays read only and locked, with no primary role file. - `keel/system/dbwatch.py` (`needs_follow`): follows a server again when it takes writes with no claim, or while the VIP is not on `wg0` (its `valid_lft` ran out). - `keel/system/dbfollow.py` (`follow`): one run at a time, under `/var/lib/keel/database/follow.lock`, so the unit and `keel database watch` never interleave between the proof and `SET read_only OFF`. - Without etcd the address has no lifetime, so the proof is weaker: #113 records it, and `docs/replication.md` says so. - `docs/replication.md`: the follow table and the rule. - `debian/changelog`, `pyproject.toml`, `keel/__init__.py`: 0.23.4. ## Tests - `tests/test_system_dbfollow.py`, `TestTheOldPrimaryAtBoot`: crash, the other node promoted at a newer epoch, the old node boots, stays read only, then follows as a replica (cloud simple, as on the real mesh); a server found writable without a proof is made read only; the holder proven again after its boot takes writes; no peer answers; the proof comes while follow waits; etcd does not answer at boot. The old code fails five of these with `SET GLOBAL read_only = OFF`. - `TestNoClaimAndNoAddress`: the state file removed on a primary with a pair record, follow keeps it read only; watch follows a writable primary whose VIP left `wg0`; follow takes its lock. - `tests/test_mesh_vipmove.py`: the state is written again when `keel vip check` adds the VIP, and not when it is there. - `tests/mariadb_netns.py`: A now takes the VIP (and follow runs) before the application's data is written, because a paired node with no claim is read only. - `tests/mariadb_netns.py` (`mariadb / trixie`): the primary's server killed, its VIP helper stopped, cut off; `keel database promote --old-primary-gone` on the replica; the old primary boots with its old claim: `@@read_only` is 1 at boot, after follow, and in every 0.5 s sample until it replicates from the new primary over TLS by GTID. ## Test plan - [ ] tests / coverage - [ ] mariadb / trixie (the new crash case) - [ ] vip, vip-upgrade, etcd, etcd-auth, mesh / trixie - [ ] package / changelog ## CI record First run (617bfbb's first version): `mariadb / trixie` failed in the new case only: the replica's I/O thread was still `Connecting` when the status was read (read only held in every sample, replica after 6.4 s). The case now waits for the I/O thread to stream, as case (a) does (amended). `vip / trixie` failed at `g_promote_b` (`etcdctl get: context deadline exceeded`), the same failure as main's run 37833124848 and run 37843439840. No job was rerun: the amended push ran all checks again, all green. Security review: H1, M2 and L1 are fixed in the amended commit; M1 is #113. Head 7cbfab1 (after the security review): every check green but `mesh / trixie`. Attempt 1 timed out in `test_a_spent_invite_is_refused` (the inviter process). I reran the job once (run 37869774880, attempt 2); it failed in another test, `test_c_joins_through_the_fallback`, the same failure as main's run 37833124848. This PR changes nothing in keel.mesh's join or invite code. #112 holds this commit plus its own, and its `mesh / trixie` passed at 132a13a. I did not rerun a second time.