database: after an automatic VIP failover with etcd, the new PostgreSQL primary takes no write until the operator runs --old-primary-gone #165

New issue

Closed

opened 2026-10-15 08:21:06 -03:00 by navigator · 0 comments

navigator commented

2026-10-15 08:21:06 -03:00

Owner

Found on real nodes on 2026-10-15 (keel 0.23.21, etcd of 3 members in 3 sites, handbook decision 0057, evidence docs/evidence/2026-10-15-br3-thin-etcd-witness.md).

What happened

A was 9054, the primary. B was 9062, its synchronous standby. A was cut off (all WireGuard packets dropped in A) at 1792062418.97.

Time	From the cut	Event
2439.65	20.7 s	B's controller: "claimed at epoch 10: the lease of epoch 9 is gone"
2450.41	31.4 s	B out of recovery (the database watch ran follow, because the follow path had failed, keel#164)
2450 to 2703		every write on B waits: `synchronous_standby_names` stays `FIRST 1 (keel)`, and no standby streams
2703.55		the operator runs `keel database promote --old-primary-gone` on B: "commits alone", 2.2 s
2705.92		the first write on B is acknowledged, 0.07 s

The VIP failover is automatic (0020, docs/vip.md: "the other node of the pair claims as above: 0020's automatic failover"). But the database takes no write until the operator acts, or until the old primary is reseeded, which also waits for the operator. Thus the client write-resume bound of 5 s (keel#148) is not met with automatic failover. 6 writes that the clients gave up on (killed after 30 s) were committed when the promote released them.

Why

The exception of 0054 (amendment 2026-10-14) empties synchronous_standby_names only after keel database promote --old-primary-gone. The controller's claim is not that statement.

Question for the maintainer

With etcd, B claims only after A's lease expired, and A dropped the VIP 10 s before that (the kernel lifetime of the address). Is the expired lease enough to take the exception, the same as --old-primary-gone? If not, docs/vip.md and 0020 must say that the failover is automatic for the VIP only, and that the writes wait for the operator.

Found on real nodes on 2026-10-15 (keel 0.23.21, etcd of 3 members in 3 sites, handbook decision 0057, evidence `docs/evidence/2026-10-15-br3-thin-etcd-witness.md`). ## What happened A was 9054, the primary. B was 9062, its synchronous standby. A was cut off (all WireGuard packets dropped in A) at 1792062418.97. | Time | From the cut | Event | | --- | --- | --- | | 2439.65 | 20.7 s | B's controller: "claimed at epoch 10: the lease of epoch 9 is gone" | | 2450.41 | 31.4 s | B out of recovery (the database watch ran follow, because the follow path had failed, keel#164) | | 2450 to 2703 | | every write on B waits: `synchronous_standby_names` stays `FIRST 1 (keel)`, and no standby streams | | 2703.55 | | the operator runs `keel database promote --old-primary-gone` on B: "commits alone", 2.2 s | | 2705.92 | | the first write on B is acknowledged, 0.07 s | The VIP failover is automatic (0020, `docs/vip.md`: "the other node of the pair claims as above: 0020's automatic failover"). But the database takes no write until the operator acts, or until the old primary is reseeded, which also waits for the operator. Thus the client write-resume bound of 5 s (keel#148) is not met with automatic failover. 6 writes that the clients gave up on (killed after 30 s) were committed when the promote released them. ## Why The exception of 0054 (amendment 2026-10-14) empties `synchronous_standby_names` only after `keel database promote --old-primary-gone`. The controller's claim is not that statement. ## Question for the maintainer With etcd, B claims only after A's lease expired, and A dropped the VIP 10 s before that (the kernel lifetime of the address). Is the expired lease enough to take the exception, the same as `--old-primary-gone`? If not, `docs/vip.md` and 0020 must say that the failover is automatic for the VIP only, and that the writes wait for the operator.