fix: a PostgreSQL server of a pair starts as a primary only on the proven holder (keel#156) #175

Merged

navigator merged 1 commit from fix/pg-start-fenced-156 into main 2026-10-15 15:02:36 -03:00

Conversation 0 Commits 1 Files changed 14 +337 -10

navigator commented 2026-10-15 09:56:37 -03:00

Owner

Copy link

Fixes #156. Stacked on #174.

A returned old primary started as a primary at boot and after a heal, and keel stopped it 3 to 4 s later. A client with synchronous_commit set to local or off got its write acknowledged in that window.

Change

keel installs a drop-in for postgresql@.service (systemd/postgresql@.service.d/keel-guard.conf). It runs keel database guard as root before each start:

The VIP here	The start
standby.signal present, or no claim yet	as it is
the newest claim is the other member's, or this node is fenced	as a standby that follows nobody: standby.signal written and the follow drop-in removed first
this node's own claim, the VIP on wg0	as the primary
this node's own claim, the VIP not on wg0 (boot)	refused; keel database follow starts it once the VIP is proven

The guard asks nobody over the network. On a node with no pair it does nothing. If the spec of a pair cannot be read, it refuses the start.

This does not start the legitimate holder as a standby and then promote it: that path hits FENCED_REFUSED, and each reboot would make a new timeline.

Tests

• RED first: tests/test_system_dbpgguard.py (8 cases), the CLI wiring, and a packaging test for the drop-in.
• netns: the harness unit runs the guard. Case (e) checks that the start at boot is refused ("not proven"), and that the old primary is fenced, dumped and recorded with no write.

Fixes #156. Stacked on #174. A returned old primary started as a primary at boot and after a heal, and keel stopped it 3 to 4 s later. A client with `synchronous_commit` set to `local` or `off` got its write acknowledged in that window. ## Change keel installs a drop-in for `postgresql@.service` (`systemd/postgresql@.service.d/keel-guard.conf`). It runs `keel database guard` as root before each start: | The VIP here | The start | | --- | --- | | `standby.signal` present, or no claim yet | as it is | | the newest claim is the other member's, or this node is fenced | as a standby that follows nobody: `standby.signal` written and the follow drop-in removed first | | this node's own claim, the VIP on wg0 | as the primary | | this node's own claim, the VIP not on wg0 (boot) | refused; `keel database follow` starts it once the VIP is proven | The guard asks nobody over the network. On a node with no pair it does nothing. If the spec of a pair cannot be read, it refuses the start. This does not start the legitimate holder as a standby and then promote it: that path hits `FENCED_REFUSED`, and each reboot would make a new timeline. ## Tests - RED first: `tests/test_system_dbpgguard.py` (8 cases), the CLI wiring, and a packaging test for the drop-in. - netns: the harness unit runs the guard. Case (e) checks that the start at boot is refused ("not proven"), and that the old primary is fenced, dumped and recorded with no write.

navigator added 6 commits 2026-10-15 09:56:37 -03:00

feat: with etcd, an expired lease states the old PostgreSQL primary gone (keel#165) ... 312e2db482

The maintainer decided keel#165 on 2026-10-15 (handbook 0054, amendment
of 2026-10-15). With etcd, an automatic failover of the PostgreSQL pair
moved the VIP, but the new primary took no write until the operator ran
keel database promote --old-primary-gone.

- The VIP controller claims after the lease of the newest claim is gone.
  When that lease expired at its due time, and the controller saw it
  alive before, the claim states that the old primary is gone. The root
  half writes /var/lib/keel/database/old-primary-gone, with the epoch of
  the claim, before it records the claim. So the follow that the claim
  starts promotes the standby and commits alone.
- A revoked lease, or a lease that the controller never saw alive (after
  a restart of the controller), states nothing. The holder that claims
  again after a revoke states nothing.
- keel database follow takes a statement with no epoch (the operator's)
  or one for the epoch of this node's newest claim. It removes a
  statement of another epoch unused.
- The root half cannot ask etcd again. It trusts the word of its own
  helper for this statement only, and it checks the claim: verified,
  this node's own, above the epoch it held, the previous holder another
  node.
- The netns case (c) now runs with no operator promote: B's controller
  claims when A's lease expired, and B commits alone.

fix: keel database promote makes the database writable before it joins the late asks (keel#169) ... b4a593d1bd

On a real pair with etcd, keel database promote --old-primary-gone made
the database writable 11.9 s and 12.9 s after the VIP moved to it. The
bound of keel#148 is 5 s.

keel vip promote bounds each ask of the gone old primary to 1.5 s, but
it joins the late asks before it returns. A dead node costs the members'
channel's 10 s connect timeout, and keel database promote observed and
followed only after that join. CI did not see it: the harness follows
the VIP's state on its own, as keel-database-follow.path does, and that
path had failed on the real nodes (keel#164).

- vippromote.promote takes `then`, a step that runs once the promote
  succeeded, before the join. keel database promote passes its database
  step there. The promote still joins the asks before it exits.
- A refused promote runs no database step. The operator's statement is
  removed only when the database step did not run.

fix: a PostgreSQL standby caught up after the reseed may take the VIP again (keel#166) ... 10ea05501f

After an automatic failover, the old holder is fenced in its VIP state.
A fenced node never claims by itself (0049). The confirmed reseed did
not clear that fence, so on real nodes the second automatic failover
did not come: the VIP stayed on no node until the operator ran keel
database promote --old-primary-gone.

- keel database follow clears the fence of a PostgreSQL standby that
  streams from the holder and received all that the holder flushed.
  The holder's flushed position comes first, from IDENTIFY_SYSTEM, then
  the standby's received position. Caught up, not only streaming: a
  commit that the holder acknowledged alone is then on this node too.
- The follow does this after a seed or a reseed, and on each run that
  finds the node a standby of the holder.
- keel database watch runs follow on a fenced PostgreSQL standby that
  streams. A node that is fenced now is cleared at its next watch run.
- A node fenced on its own claim stays fenced.
- The netns test checks that the reseeded old primary is not fenced
  within 3 watch runs.

fix: the VIP's lock no longer starts keel-database-follow.path (keel#164) ... 36c447fd76

With etcd, keel-database-follow.path (PathChanged=/var/lib/keel/vip)
started keel database follow every 1 to 2 s on both nodes of the pair.
On the replica it failed with unit-start-limit-hit in a few seconds,
and after that only the watch, every 30 s, saw a VIP move.

The cause: vipstate.locked() opened /var/lib/keel/vip/lock for writing.
The VIP controller takes that lock at each turn: take() every 1 s on
each node, and carry_held() at each confirmed renewal, every 2 s on the
holder. The close of a file opened for writing is IN_CLOSE_WRITE, an
event of PathChanged=, even when nothing was written. The numbers match
the evidence: a run every 2.2 s on the holder, 5 runs in 7 s on the
replica.

- The lock file is opened read only. flock needs no write access, and
  the close is then IN_CLOSE_NOWRITE. The path does not change, so a
  helper of the older version shares the lock during the upgrade.
- tests/pathwatch.py watches a directory with systemd's PathChanged
  mask. A unit test runs the controllers of a steady pair for 10 s and
  sees no event; a move of the VIP still makes one.
- The netns harness follows on these events instead of the mtime of the
  files, as systemd does. The mtime did not change, so CI missed the
  loop. The netns test checks that a steady pair starts at most 1
  follow in 20 s.

fix: the VIP rides out an etcd re-election, and the watch starts a stopped holder (keel#167, keel#168) ... 5a1bdbedb0

keel#167: a cut of the site that held etcd's leader made the VIP holder
drop the VIP for 2 to 4 s on real nodes. The new leader came 7.4 s after
the cut. The kernel ended the address int(10 - age) - 1 s after the
carry, so about 8 s after the last renewal that the majority confirmed.

- RELEASE_AFTER is 15 s, not 10 s. The kernel now keeps the address up
  to 14 s after that renewal, which covers one election of 5 to 10 s
  and a renewal. The release stays 5 s inside the TTL of 20 s, which
  etcd counts from its receipt of the renewal, so a holder cut off from
  the majority has dropped the VIP before any claim is possible.
- With the longer grace, a lease that was revoked early could look
  expired once the node waited until its due time. The controller now
  takes a lease as expired only when it first saw it gone at its due
  time (keel#165).
- The netns tests read RELEASE_AFTER from keel, not a copy of 10 s.

The 111 s with no etcd leader under netem is not changed here: see the
analysis in keel#167.

keel#168: during that time the watch stopped the holder's server ("not
proven, the VIP is not on wg0"). The VIP came back, but the server
stayed stopped: the watch followed only a server that answered.

- keel database watch now runs keel database follow on the holder when
  its server does not answer. Follow starts it once the VIP is on wg0.

fix: a PostgreSQL server of a pair starts as a primary only on the proven holder (keel#156) ... 66feb03725

A returned old primary started as a primary at boot and after a heal,
and keel stopped it 3 to 4 s later. PostgreSQL cannot be made read only
while it runs. A default commit waited for the standby, but a client
with synchronous_commit set to local or off got its write acknowledged
in that window, and the pair does not hold it.

keel now installs a drop-in for postgresql@.service. It runs keel
database guard as root before each start:

- with standby.signal, or before the pair's first claim: the server
  starts as it is;
- the newest claim is the other member's, or this node is fenced: the
  guard writes standby.signal and removes the follow drop-in, so the
  server starts as a standby that follows nobody (the heal window);
- this node's own claim, and the VIP on wg0: the server starts as the
  primary;
- this node's own claim, the VIP not on wg0 (the boot window): the
  start is refused. keel database follow starts the server once the
  VIP is proven, as before.

The guard asks nobody over the network. On a node with no pair, or no
PostgreSQL in its spec, it does nothing. If the spec of a pair cannot
be read, it refuses the start.

The netns harness runs the guard in its server unit. Case (e) checks
that the start at boot is refused and that the old primary is fenced,
dumped and recorded with no write.

navigator force-pushed fix/pg-start-fenced-156 from 66feb03725 to 8b2fb1ea84 2026-10-15 10:30:14 -03:00 Compare

navigator force-pushed fix/pg-start-fenced-156 from 8b2fb1ea84 to f4e09f48fc 2026-10-15 11:17:10 -03:00 Compare

navigator force-pushed fix/pg-start-fenced-156 from f4e09f48fc to 5a19d0d569 2026-10-15 12:03:25 -03:00 Compare

navigator force-pushed fix/pg-start-fenced-156 from 5a19d0d569 to efcb17a637 2026-10-15 12:45:27 -03:00 Compare

navigator force-pushed fix/pg-start-fenced-156 from efcb17a637 to ce640b1499 2026-10-15 13:30:24 -03:00 Compare

navigator referenced this pull request 2026-10-15 13:54:27 -03:00

feat: keel verify trusts a signature by main or images (decision 0055) #176

navigator force-pushed fix/pg-start-fenced-156 from ce640b1499 to 265ff5b47c 2026-10-15 14:11:57 -03:00 Compare

navigator force-pushed fix/pg-start-fenced-156 from 265ff5b47c to c56252b959 2026-10-15 14:31:56 -03:00 Compare

navigator referenced this pull request 2026-10-15 14:32:15 -03:00

ci: flaky tests cost about 1 h per PR (mesh channel TLS EOF, vip carry wait, etcd deadline) #177

navigator merged commit 76f06c01fd into main 2026-10-15 15:02:36 -03:00

navigator referenced this pull request from a commit 2026-10-15 15:02:36 -03:00

Merge pull request 'fix: a PostgreSQL server of a pair starts as a primary only on the proven holder (keel#156)' (#175) from fix/pg-start-fenced-156 into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

accessibility

Barrier affecting people with disabilities

  

bug

Something isn't working

  

documentation

Improvements or additions to documentation

  

duplicate

This issue or pull request already exists

  

enhancement

New feature or request

  

good first issue

Good for newcomers

  

help wanted

Extra attention is needed

  

invalid

This doesn't seem right

  

question

Further information is requested

  

wontfix

This will not be worked on

No labels

accessibility

bug

documentation

duplicate

enhancement

good first issue

help wanted

invalid

question

wontfix

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

Keel-Linux/keel!175

No description provided.