fix: keel database promote makes the database writable before it joins the late asks (keel#169) #171

Merged

navigator merged 1 commit from fix/pg-promote-follow-before-join-169 into main 2026-10-15 11:44:41 -03:00

Conversation 0 Commits 1 Files changed 8 +102 -29

navigator commented 2026-10-15 09:32:29 -03:00

Owner

Copy link

Fixes #169. Stacked on #170 (keel#165): merge #170 first.

On a real pair with etcd, keel database promote --old-primary-gone made the database writable 11.9 s and 12.9 s after the VIP moved (bound 5 s, keel#148).

Where the 12 s went

keel vip promote bounds each ask of the gone old primary to 1.5 s (keel#108), but it joins the late asks before it returns (vipnode.finished). A dead node costs the members' channel's 10 s connect timeout. keel database promote observed and followed only after that join: the VIP at 2927.87, out of recovery at 2939.45, about 10 s plus 1.5 s. CI did not see it, because the harness follows the VIP's state by itself, as keel-database-follow.path does, and that path had failed on the real nodes (keel#164).

Change

• vippromote.promote takes then, a step that runs once the promote succeeded, before the join. keel database promote passes its database step there. The promote still joins the asks before it exits.
• A refused promote runs no database step. The operator's statement is removed only when the database step did not run.

Tests

RED first, in tests/test_mesh_vipetcd.py: the database step runs within 2 s while the ask of the old primary takes 3 s, and the promote returns after the ask; a refused promote runs no database step.

Fixes #169. Stacked on #170 (keel#165): merge #170 first. On a real pair with etcd, `keel database promote --old-primary-gone` made the database writable 11.9 s and 12.9 s after the VIP moved (bound 5 s, keel#148). ## Where the 12 s went `keel vip promote` bounds each ask of the gone old primary to 1.5 s (keel#108), but it joins the late asks before it returns (`vipnode.finished`). A dead node costs the members' channel's 10 s connect timeout. `keel database promote` observed and followed only after that join: the VIP at 2927.87, out of recovery at 2939.45, about 10 s plus 1.5 s. CI did not see it, because the harness follows the VIP's state by itself, as `keel-database-follow.path` does, and that path had failed on the real nodes (keel#164). ## Change - `vippromote.promote` takes `then`, a step that runs once the promote succeeded, before the join. `keel database promote` passes its database step there. The promote still joins the asks before it exits. - A refused promote runs no database step. The operator's statement is removed only when the database step did not run. ## Tests RED first, in `tests/test_mesh_vipetcd.py`: the database step runs within 2 s while the ask of the old primary takes 3 s, and the promote returns after the ask; a refused promote runs no database step.

navigator added 2 commits 2026-10-15 09:32:29 -03:00

feat: with etcd, an expired lease states the old PostgreSQL primary gone (keel#165) ... 31cae50205

The maintainer decided keel#165 on 2026-10-15 (handbook 0054, amendment
of 2026-10-15). With etcd, an automatic failover of the PostgreSQL pair
moved the VIP, but the new primary took no write until the operator ran
keel database promote --old-primary-gone.

- The VIP controller claims after the lease of the newest claim is gone.
  When that lease expired at its due time, and the controller saw it
  alive before, the claim states that the old primary is gone. The root
  half writes /var/lib/keel/database/old-primary-gone, with the epoch of
  the claim, before it records the claim. So the follow that the claim
  starts promotes the standby and commits alone.
- A revoked lease, or a lease that the controller never saw alive (after
  a restart of the controller), states nothing. The holder that claims
  again after a revoke states nothing.
- keel database follow takes a statement with no epoch (the operator's)
  or one for the epoch of this node's newest claim. It removes a
  statement of another epoch unused.
- The root half cannot ask etcd again. It trusts the word of its own
  helper for this statement only, and it checks the claim: verified,
  this node's own, above the epoch it held, the previous holder another
  node.
- The netns case (c) now runs with no operator promote: B's controller
  claims when A's lease expired, and B commits alone.

fix: keel database promote makes the database writable before it joins the late asks (keel#169) ... 34190e4fa6

On a real pair with etcd, keel database promote --old-primary-gone made
the database writable 11.9 s and 12.9 s after the VIP moved to it. The
bound of keel#148 is 5 s.

keel vip promote bounds each ask of the gone old primary to 1.5 s, but
it joins the late asks before it returns. A dead node costs the members'
channel's 10 s connect timeout, and keel database promote observed and
followed only after that join. CI did not see it: the harness follows
the VIP's state on its own, as keel-database-follow.path does, and that
path had failed on the real nodes (keel#164).

- vippromote.promote takes `then`, a step that runs once the promote
  succeeded, before the join. keel database promote passes its database
  step there. The promote still joins the asks before it exits.
- A refused promote runs no database step. The operator's statement is
  removed only when the database step did not run.

navigator force-pushed fix/pg-promote-follow-before-join-169 from 34190e4fa6 to b4a593d1bd 2026-10-15 09:52:03 -03:00 Compare

navigator referenced this pull request 2026-10-15 09:56:35 -03:00

fix: a PostgreSQL standby caught up after the reseed may take the VIP again (keel#166) #172

navigator force-pushed fix/pg-promote-follow-before-join-169 from b4a593d1bd to 61e30f9b76 2026-10-15 10:30:10 -03:00 Compare

navigator force-pushed fix/pg-promote-follow-before-join-169 from 61e30f9b76 to bec6e3ccc4 2026-10-15 11:17:06 -03:00 Compare

navigator merged commit 0091d862c5 into main 2026-10-15 11:44:41 -03:00

navigator referenced this pull request from a commit 2026-10-15 11:44:41 -03:00

Merge pull request 'fix: keel database promote makes the database writable before it joins the late asks (keel#169)' (#171) from fix/pg-promote-follow-before-join-169 into main

navigator referenced this pull request 2026-10-15 14:32:15 -03:00

ci: flaky tests cost about 1 h per PR (mesh channel TLS EOF, vip carry wait, etcd deadline) #177

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

accessibility

Barrier affecting people with disabilities

  

bug

Something isn't working

  

documentation

Improvements or additions to documentation

  

duplicate

This issue or pull request already exists

  

enhancement

New feature or request

  

good first issue

Good for newcomers

  

help wanted

Extra attention is needed

  

invalid

This doesn't seem right

  

question

Further information is requested

  

wontfix

This will not be worked on

No labels

accessibility

bug

documentation

duplicate

enhancement

good first issue

help wanted

invalid

question

wontfix

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

Keel-Linux/keel!171

No description provided.