database: a planned keel database promote leaves the old PostgreSQL primary diverged, by its shutdown checkpoint #163

New issue

Open

opened 2026-10-15 08:05:05 -03:00 by navigator · 0 comments

navigator commented

2026-10-15 08:05:05 -03:00

Owner

Found on real nodes on 2026-10-15 (keel 0.23.21, etcd of 3 members, handbook decision 0057).

What happened

A was 9062, the primary, with FIRST 1 (keel). B was 9054, the synchronous standby. A co-located writer on A wrote every 0.35 s. On B, at 1792062057.44:

$ keel database promote          # no --old-primary-gone
released by fd1a:bb4c:b407::fc0a
this node holds fd1a:bb4c:b407::ffff:1 at epoch 9, by etcd's lease ...
database promote: 1 change(s), 0 failed
rc=0 (4.0 s)

Timeline:

Time	Node	Event
2058.04	A	released the VIP
2059.28	A	`Stopping postgresql@17-main` (the fence restart)
2059.89	B	"drained and promoted"
2063.20	A	`Stopped postgresql@17-main`
2066.98	A	fenced, dump kept, "refused: diverged (timeline 2 against the holder's 3)"

B's 00000003.history: timeline 2 ends at 0/385CAB0. A's pg_controldata: latest checkpoint location 0/385CAB0, timeline 2. Thus A holds 1 record that B never got: the shutdown checkpoint that A wrote at its stop, after B promoted.

All 16 writes that A acknowledged (100000 to 100015) are on B. No data is lost. But A needs a confirmed reseed (keel spec apply --system-only --destroy-local-database, 10.8 s). Until then, B has no standby, and every commit on B waits (0054). The writer on B got 4 writes killed after 30 s each.

Expected

A planned promote waits until the old primary has stopped and the standby has received the old primary's last WAL record (its shutdown checkpoint), then promotes. Then the old primary follows the new one with no divergence and no reseed. This is the path of docs/vip.md, "Upgrading a pair without downtime".

Found on real nodes on 2026-10-15 (keel 0.23.21, etcd of 3 members, handbook decision 0057). ## What happened A was 9062, the primary, with `FIRST 1 (keel)`. B was 9054, the synchronous standby. A co-located writer on A wrote every 0.35 s. On B, at 1792062057.44: ``` $ keel database promote # no --old-primary-gone released by fd1a:bb4c:b407::fc0a this node holds fd1a:bb4c:b407::ffff:1 at epoch 9, by etcd's lease ... database promote: 1 change(s), 0 failed rc=0 (4.0 s) ``` Timeline: | Time | Node | Event | | --- | --- | --- | | 2058.04 | A | released the VIP | | 2059.28 | A | `Stopping postgresql@17-main` (the fence restart) | | 2059.89 | B | "drained and promoted" | | 2063.20 | A | `Stopped postgresql@17-main` | | 2066.98 | A | fenced, dump kept, "refused: diverged (timeline 2 against the holder's 3)" | B's `00000003.history`: timeline 2 ends at `0/385CAB0`. A's `pg_controldata`: latest checkpoint location `0/385CAB0`, timeline 2. Thus A holds 1 record that B never got: the shutdown checkpoint that A wrote at its stop, after B promoted. All 16 writes that A acknowledged (100000 to 100015) are on B. No data is lost. But A needs a confirmed reseed (`keel spec apply --system-only --destroy-local-database`, 10.8 s). Until then, B has no standby, and every commit on B waits (0054). The writer on B got 4 writes killed after 30 s each. ## Expected A planned promote waits until the old primary has stopped and the standby has received the old primary's last WAL record (its shutdown checkpoint), then promotes. Then the old primary follows the new one with no divergence and no reseed. This is the path of `docs/vip.md`, "Upgrading a pair without downtime".