feat: with etcd, an expired lease states the old PostgreSQL primary gone (keel#165) #170

Merged

navigator merged 1 commit from fix/pg-expired-lease-alone-165 into main

2026-10-15 10:20:59 -03:00

navigator commented

2026-10-15 09:29:05 -03:00

Owner

Fixes #165. The maintainer decided it on 2026-10-15 ("sim"); handbook 0054, amendment of 2026-10-15 (handbook#79).

With etcd, an automatic failover moved the VIP, but the new PostgreSQL primary took no write until the operator ran keel database promote --old-primary-gone (evidence docs/evidence/2026-10-15-br3-thin-etcd-witness.md, section 5).

Change

The VIP controller states that the old primary is gone when it claims because the lease of the newest claim expired at its due time, and it saw that lease alive before. The root half writes /var/lib/keel/database/old-primary-gone, with the epoch of the claim, before it records the claim. The follow that the claim starts then promotes the standby and commits alone.
A revoked lease, or a lease that the controller never saw alive (after a restart of the controller), states nothing. The holder that claims again after a revoke states nothing.
keel database follow takes a statement with no epoch (the operator's) or one for the epoch of the newest claim. It removes a statement of another epoch unused.
Trust boundary: the root half cannot ask etcd again, so it trusts its own helper for this statement only. It checks the claim: verified, this node's own, above the epoch it held, the previous holder another node.

Tests

RED first: TestTheOldPrimaryGone in tests/test_mesh_vipetcd.py (expired, revoked, never seen alive, reclaim, the bridge), and 2 tests in tests/test_system_dbpgfollow.py (the statement of this claim, a stale statement).
netns: case (c) of tests/postgres_netns.py now runs with no operator promote. B's controller claims when A's lease expired, B commits alone, and the bound of 5 s from the VIP holds.

Fixes #165. The maintainer decided it on 2026-10-15 ("sim"); handbook 0054, amendment of 2026-10-15 (handbook#79). With etcd, an automatic failover moved the VIP, but the new PostgreSQL primary took no write until the operator ran `keel database promote --old-primary-gone` (evidence `docs/evidence/2026-10-15-br3-thin-etcd-witness.md`, section 5). ## Change - The VIP controller states that the old primary is gone when it claims because the lease of the newest claim expired at its due time, and it saw that lease alive before. The root half writes `/var/lib/keel/database/old-primary-gone`, with the epoch of the claim, before it records the claim. The follow that the claim starts then promotes the standby and commits alone. - A revoked lease, or a lease that the controller never saw alive (after a restart of the controller), states nothing. The holder that claims again after a revoke states nothing. - `keel database follow` takes a statement with no epoch (the operator's) or one for the epoch of the newest claim. It removes a statement of another epoch unused. - Trust boundary: the root half cannot ask etcd again, so it trusts its own helper for this statement only. It checks the claim: verified, this node's own, above the epoch it held, the previous holder another node. ## Tests - RED first: `TestTheOldPrimaryGone` in `tests/test_mesh_vipetcd.py` (expired, revoked, never seen alive, reclaim, the bridge), and 2 tests in `tests/test_system_dbpgfollow.py` (the statement of this claim, a stale statement). - netns: case (c) of `tests/postgres_netns.py` now runs with no operator promote. B's controller claims when A's lease expired, B commits alone, and the bound of 5 s from the VIP holds.

navigator added 1 commit

2026-10-15 09:29:05 -03:00

feat: with etcd, an expired lease states the old PostgreSQL primary gone (keel#165)

security / security (pull_request) Successful in 1m15s

Details

tests / etcd-auth (pull_request) Successful in 5m5s

Details

tests / etcd (pull_request) Successful in 7m13s

Details

tests / vip (pull_request) Successful in 7m41s

Details

tests / vip-upgrade (pull_request) Failing after 7m46s

Details

tests / package (pull_request) Successful in 5s

Details

tests / mesh (pull_request) Successful in 11m9s

Details

tests / mariadb-simple (pull_request) Successful in 4m36s

Details

tests / mariadb (pull_request) Successful in 5m39s

Details

tests / postgres (pull_request) Failing after 10m13s

Details

tests / tests (pull_request) Has been cancelled

Details

31cae50205

The maintainer decided keel#165 on 2026-10-15 (handbook 0054, amendment
of 2026-10-15). With etcd, an automatic failover of the PostgreSQL pair
moved the VIP, but the new primary took no write until the operator ran
keel database promote --old-primary-gone.

- The VIP controller claims after the lease of the newest claim is gone.
  When that lease expired at its due time, and the controller saw it
  alive before, the claim states that the old primary is gone. The root
  half writes /var/lib/keel/database/old-primary-gone, with the epoch of
  the claim, before it records the claim. So the follow that the claim
  starts promotes the standby and commits alone.
- A revoked lease, or a lease that the controller never saw alive (after
  a restart of the controller), states nothing. The holder that claims
  again after a revoke states nothing.
- keel database follow takes a statement with no epoch (the operator's)
  or one for the epoch of this node's newest claim. It removes a
  statement of another epoch unused.
- The root half cannot ask etcd again. It trusts the word of its own
  helper for this statement only, and it checks the claim: verified,
  this node's own, above the epoch it held, the previous holder another
  node.
- The netns case (c) now runs with no operator promote: B's controller
  claims when A's lease expired, and B commits alone.