vip: with etcd, the automatic failover leaves the VIP on no node: the statement of keel#165 cannot be written in the sandbox of keel-vip.service #178

New issue

Closed

opened 2026-10-15 15:19:18 -03:00 by navigator · 0 comments

navigator commented

2026-10-15 15:19:18 -03:00

Owner

Found on real nodes on 2026-10-15 (keel 0.23.27, the PostgreSQL pair 9054 and 9062 with the etcd witness 9071; evidence in the handbook, docs/evidence/2026-10-15-pair-etcd-failover-real.md).

What happened

Case (c) at base: 9054, the primary, was cut off at 1792087979.98. The VIP controller of 9062 claimed epoch 14 in etcd when the lease of epoch 13 expired. Then:

1792088013.439298 postgresql-br2 python3[370035]: vip: [Errno 30] Read-only file system: '/var/lib/keel/database/old-primary-gone.new'

The root half of keel-vip.service writes the statement of keel#165 to /var/lib/keel/database/old-primary-gone. The unit has ProtectSystem=strict and ReadWritePaths=/var/lib/keel/vip /etc/wireguard, so the write fails. Ops.hold raised before it recorded the claim. The next turn took the claim from the counter with take(), which keeps no lease for this node's own claim. The controller renews and carries only claims with a lease. Thus 9062 held epoch 14 in etcd and in its state, with no lease, and no node carried the VIP. 9062 stayed in recovery. The pair took no write until the operator ran keel database promote --old-primary-gone on 9054.

CI did not see it: the netns harness runs the helper with ReadWritePaths on the whole root of the node.

Expected

The controller writes its statement inside /var/lib/keel/vip, the state directory of the unit.
A statement that cannot be written never stops the claim: the claim is recorded with its lease, and the VIP is carried.
take() of this node's own claim from the counter keeps the lease that is signed into it, so a claim that etcd holds is renewed and carried.
The harness runs the helper with the ReadWritePaths of keel-vip.service.

Found on real nodes on 2026-10-15 (keel 0.23.27, the PostgreSQL pair 9054 and 9062 with the etcd witness 9071; evidence in the handbook, `docs/evidence/2026-10-15-pair-etcd-failover-real.md`). ## What happened Case (c) at base: 9054, the primary, was cut off at 1792087979.98. The VIP controller of 9062 claimed epoch 14 in etcd when the lease of epoch 13 expired. Then: ``` 1792088013.439298 postgresql-br2 python3[370035]: vip: [Errno 30] Read-only file system: '/var/lib/keel/database/old-primary-gone.new' ``` The root half of keel-vip.service writes the statement of keel#165 to `/var/lib/keel/database/old-primary-gone`. The unit has `ProtectSystem=strict` and `ReadWritePaths=/var/lib/keel/vip /etc/wireguard`, so the write fails. `Ops.hold` raised before it recorded the claim. The next turn took the claim from the counter with `take()`, which keeps no lease for this node's own claim. The controller renews and carries only claims with a lease. Thus 9062 held epoch 14 in etcd and in its state, with no lease, and no node carried the VIP. 9062 stayed in recovery. The pair took no write until the operator ran `keel database promote --old-primary-gone` on 9054. CI did not see it: the netns harness runs the helper with `ReadWritePaths` on the whole root of the node. ## Expected - The controller writes its statement inside `/var/lib/keel/vip`, the state directory of the unit. - A statement that cannot be written never stops the claim: the claim is recorded with its lease, and the VIP is carried. - `take()` of this node's own claim from the counter keeps the lease that is signed into it, so a claim that etcd holds is renewed and carried. - The harness runs the helper with the `ReadWritePaths` of keel-vip.service.