vip: the confirmed reseed does not clear the VIP fence, so automatic failover works only once #166

New issue

Closed

opened 2026-10-15 08:21:07 -03:00 by navigator · 0 comments

navigator commented

2026-10-15 08:21:07 -03:00

Owner

Found on real nodes on 2026-10-15 (keel 0.23.21, etcd of 3 members in 3 sites, handbook decision 0057, evidence docs/evidence/2026-10-15-br3-thin-etcd-witness.md).

What happened

9054 was the holder. It was cut off. Its controller dropped the VIP ("dropped, its lease is gone"), and the state became fenced: true. 9062 claimed epoch 10.
9054 rejoined by the confirmed reseed (keel spec apply --system-only --destroy-local-database). It streamed as the synchronous standby.
9062 (the holder) was cut off at 1792062863.24. 9062 dropped the VIP at 1792062870.70 (7.5 s, the kernel lifetime). 9054 did not claim. 60 s later the VIP was on no node, until the operator ran keel database promote --old-primary-gone on 9054 (VIP at 1792062927.87).

fail_over() returns when held.fenced. vipnode.take() keeps fenced from the earlier state when it records a newer claim. Nothing clears it except this node's own claim. keel vip status on such a node says: "fenced: this node lost it while cut off or away, and takes it again only by keel vip promote here".

The same happened to 9062 after it rejoined: fenced= True in its state, after the reseed.

Expected

0049 says that an old primary must never claim again by itself. After the confirmed reseed, the node holds nothing the holder lacks, and it is the synchronous standby of the current claim. The reseed (or the first time that the node streams from the holder of the newest claim) clears the fence, so that the next failover is automatic again.

Found on real nodes on 2026-10-15 (keel 0.23.21, etcd of 3 members in 3 sites, handbook decision 0057, evidence `docs/evidence/2026-10-15-br3-thin-etcd-witness.md`). ## What happened 1. 9054 was the holder. It was cut off. Its controller dropped the VIP ("dropped, its lease is gone"), and the state became `fenced: true`. 9062 claimed epoch 10. 2. 9054 rejoined by the confirmed reseed (`keel spec apply --system-only --destroy-local-database`). It streamed as the synchronous standby. 3. 9062 (the holder) was cut off at 1792062863.24. 9062 dropped the VIP at 1792062870.70 (7.5 s, the kernel lifetime). 9054 did not claim. 60 s later the VIP was on no node, until the operator ran `keel database promote --old-primary-gone` on 9054 (VIP at 1792062927.87). `fail_over()` returns when `held.fenced`. `vipnode.take()` keeps `fenced` from the earlier state when it records a newer claim. Nothing clears it except this node's own claim. `keel vip status` on such a node says: "fenced: this node lost it while cut off or away, and takes it again only by keel vip promote here". The same happened to 9062 after it rejoined: `fenced= True` in its state, after the reseed. ## Expected 0049 says that an old primary must never claim again by itself. After the confirmed reseed, the node holds nothing the holder lacks, and it is the synchronous standby of the current claim. The reseed (or the first time that the node streams from the holder of the newest claim) clears the fence, so that the next failover is automatic again.