PostgreSQL pair: a returned old primary acknowledges writes with synchronous_commit = local before its boot follow stops it #156

New issue

Closed

opened 2026-10-14 18:27:18 -03:00 by marcos-mendez · 1 comment

marcos-mendez commented

2026-10-14 18:27:18 -03:00

(Migrated from github.com)

Found on real nodes on 2026-10-14 (keel 0.23.18, PostgreSQL VIP pair, case (e)). The evidence is in the handbook, docs/evidence/2026-10-14-postgresql-pair-real.md, section 9.

What occurs

A returned old primary boots as a primary. Its boot follow stops the server, but only about 17 s later (CT 9054: started 1792012202.89, stopped 1792012218.25; CT 9062: 1792010721.31 to 1792010738.72). In that window the server takes writes. The pair's synchronous_standby_names = 'FIRST 1 (keel)' makes a default commit wait, so a default client gets no ack. But a client may set synchronous_commit = local (or off) for its session, and then its commits are acknowledged.

Test on CT 9054: a writer sent SET synchronous_commit = local before each INSERT, from the boot.

29 INSERTs acknowledged in the boot window (before 1792012218.25).
32 more acknowledged after the watch started the server again (1792012226), which is keel#154.

The fence later put all 61 rows in the dump. The new primary does not have them, so the clients saw 61 acknowledged writes that the pair does not hold.

Why this is separate from keel#154

The boot window is part of the design of case (e) in 0054 and docs/replication-postgresql.md ("Its server starts as a primary, and its boot follow, with its own old claim not proven, stops it"). It exists with etcd and in CI too. A fix for keel#154 closes only the later windows. The promise "the old primary accepts no write" holds only for clients that keep the default synchronous_commit.

Possible directions (not tested)

Keep a node whose own claim is not proven from starting as a primary at boot (for example, standby.signal before the server starts, or the follow ordered before postgresql@*.service).
Or document that clients must not lower synchronous_commit on a pair.

Found on real nodes on 2026-10-14 (keel 0.23.18, PostgreSQL VIP pair, case (e)). The evidence is in the handbook, `docs/evidence/2026-10-14-postgresql-pair-real.md`, section 9. ## What occurs A returned old primary boots as a primary. Its boot follow stops the server, but only about 17 s later (CT 9054: started 1792012202.89, stopped 1792012218.25; CT 9062: 1792010721.31 to 1792010738.72). In that window the server takes writes. The pair's `synchronous_standby_names = 'FIRST 1 (keel)'` makes a default commit wait, so a default client gets no ack. But a client may set `synchronous_commit = local` (or `off`) for its session, and then its commits are acknowledged. Test on CT 9054: a writer sent `SET synchronous_commit = local` before each INSERT, from the boot. - 29 INSERTs acknowledged in the boot window (before 1792012218.25). - 32 more acknowledged after the watch started the server again (1792012226), which is keel#154. The fence later put all 61 rows in the dump. The new primary does not have them, so the clients saw 61 acknowledged writes that the pair does not hold. ## Why this is separate from keel#154 The boot window is part of the design of case (e) in 0054 and `docs/replication-postgresql.md` ("Its server starts as a primary, and its boot follow, with its own old claim not proven, stops it"). It exists with etcd and in CI too. A fix for keel#154 closes only the later windows. The promise "the old primary accepts no write" holds only for clients that keep the default `synchronous_commit`. ## Possible directions (not tested) - Keep a node whose own claim is not proven from starting as a primary at boot (for example, `standby.signal` before the server starts, or the follow ordered before `postgresql@*.service`). - Or document that clients must not lower `synchronous_commit` on a pair.

navigator commented

2026-10-15 08:42:25 -03:00

Owner

Found on real nodes on 2026-10-15 (keel 0.23.21, etcd of 3 members in 3 sites: 9054 BR1, 9062 BR2, witness 9071 BR3; handbook decision 0057, evidence docs/evidence/2026-10-15-br3-thin-etcd-witness.md).

The same window comes after a partition heals, not only at boot. The old primary was cut off (WireGuard dropped), lost the VIP, and its watch stopped its server. After the heal:

Run	Heal	Newer claim taken	Server started	Server stopped (fence restart)	Writable window
base, 9062	1792063003.84	3005.76	3006.21	3010.02	3.8 s
netem 250 ms, 9054	1792064300.59	4309.52	4316.20	4319.09	2.9 s

In both runs the server started as a primary after the node had recorded the newer claim. No writer wrote in the window, because the test writer writes only while the node carries the VIP. A client on the local socket with synchronous_commit = local could get an acknowledged write in it.

Found on real nodes on 2026-10-15 (keel 0.23.21, etcd of 3 members in 3 sites: 9054 BR1, 9062 BR2, witness 9071 BR3; handbook decision 0057, evidence `docs/evidence/2026-10-15-br3-thin-etcd-witness.md`). The same window comes after a partition heals, not only at boot. The old primary was cut off (WireGuard dropped), lost the VIP, and its watch stopped its server. After the heal: | Run | Heal | Newer claim taken | Server started | Server stopped (fence restart) | Writable window | | --- | --- | --- | --- | --- | --- | | base, 9062 | 1792063003.84 | 3005.76 | 3006.21 | 3010.02 | 3.8 s | | netem 250 ms, 9054 | 1792064300.59 | 4309.52 | 4316.20 | 4319.09 | 2.9 s | In both runs the server started as a primary after the node had recorded the newer claim. No writer wrote in the window, because the test writer writes only while the node carries the VIP. A client on the local socket with `synchronous_commit = local` could get an acknowledged write in it.