database: the server on the VIP holder stays stopped after the VIP comes back, until keel database follow runs by hand #168

New issue

Closed

opened 2026-10-15 08:42:25 -03:00 by navigator · 0 comments

navigator commented

2026-10-15 08:42:25 -03:00

Owner

Found on real nodes on 2026-10-15 (keel 0.23.21, etcd of 3 members in 3 sites: 9054 BR1, 9062 BR2, witness 9071 BR3; handbook decision 0057, evidence docs/evidence/2026-10-15-br3-thin-etcd-witness.md).

What happened

During the 111 s with no etcd leader (keel#167), the holder 9054 lost the VIP. Its watch stopped the server at 1792063852.35: "stopped: this node's claim of fd1a:bb4c:b407::ffff:1 is not proven (the VIP is not on wg0)".

The cut healed at 1792063865.70. etcd elected a leader at 1792063913.27. The VIP came back to 9054 at the same epoch 11, with the same lease (keel vip status: carried here: yes).

The server stayed stopped. The watch runs at 1792063935.88, 1792063970.97 and 1792064006.16 logged only "the synchronous commit is not readable" (no socket). They did not say "does not match the VIP's role", and they did not start the server. keel-database-follow.path was failed (keel#164).

The restore: keel database follow on 9054 at 1792064029.50. It said "unchanged (the primary, writable)" and started the server (Started postgresql@17-main at 1792064032.37). The pair had no writable server for 180 s, 166.7 s of it after the heal.

Expected

The watch sees a stopped server on the node that carries the VIP as a mismatch, and follow starts it.

Found on real nodes on 2026-10-15 (keel 0.23.21, etcd of 3 members in 3 sites: 9054 BR1, 9062 BR2, witness 9071 BR3; handbook decision 0057, evidence `docs/evidence/2026-10-15-br3-thin-etcd-witness.md`). ## What happened During the 111 s with no etcd leader (keel#167), the holder 9054 lost the VIP. Its watch stopped the server at 1792063852.35: "stopped: this node's claim of fd1a:bb4c:b407::ffff:1 is not proven (the VIP is not on wg0)". The cut healed at 1792063865.70. etcd elected a leader at 1792063913.27. The VIP came back to 9054 at the same epoch 11, with the same lease (`keel vip status`: carried here: yes). The server stayed stopped. The watch runs at 1792063935.88, 1792063970.97 and 1792064006.16 logged only "the synchronous commit is not readable" (no socket). They did not say "does not match the VIP's role", and they did not start the server. `keel-database-follow.path` was failed (keel#164). The restore: `keel database follow` on 9054 at 1792064029.50. It said "unchanged (the primary, writable)" and started the server (`Started postgresql@17-main` at 1792064032.37). The pair had no writable server for 180 s, 166.7 s of it after the heal. ## Expected The watch sees a stopped server on the node that carries the VIP as a mismatch, and follow starts it.