fix: the statement of an expired lease lives in the VIP's state directory (keel#178) #179

Merged

navigator merged 1 commit from fix/vip-gone-note-in-state-dir-178 into main 2026-10-15 17:01:38 -03:00

Conversation 0 Commits 1 Files changed 12 +145 -41

navigator commented 2026-10-15 15:23:15 -03:00

Owner

Copy link

Fixes #178.

On real nodes (keel 0.23.27), the automatic failover with etcd left the VIP on no node. The root half of keel-vip.service wrote the statement of #165 to /var/lib/keel/database/old-primary-gone. The unit has ProtectSystem=strict and ReadWritePaths=/var/lib/keel/vip /etc/wireguard, so the write failed ([Errno 30] Read-only file system), and Ops.hold raised before it recorded the claim. The next turn took the claim from the counter with take(), which kept no lease for this node's own claim. The controller carries only a claim with its lease.

Change

• The controller's statement is /var/lib/keel/vip/old-primary-gone. The operator's stays under /var/lib/keel/database. keel database follow reads and removes both.
• A statement that cannot be written is said, and the claim is recorded with its lease all the same.
• take() of this node's own claim keeps the lease that is signed into it.
• The netns harness runs the VIP helper with the write paths of keel-vip.service, not the whole root of the node. That difference is why CI missed it.

Tests

RED first in tests/test_mesh_vipetcd.py: the note is in the state directory; a statement that cannot be written does not stop the claim (the VIP is carried, the lease recorded); a node's own claim taken from the counter keeps its lease and is carried.

Fixes #178. On real nodes (keel 0.23.27), the automatic failover with etcd left the VIP on no node. The root half of keel-vip.service wrote the statement of #165 to `/var/lib/keel/database/old-primary-gone`. The unit has `ProtectSystem=strict` and `ReadWritePaths=/var/lib/keel/vip /etc/wireguard`, so the write failed (`[Errno 30] Read-only file system`), and `Ops.hold` raised before it recorded the claim. The next turn took the claim from the counter with `take()`, which kept no lease for this node's own claim. The controller carries only a claim with its lease. ## Change - The controller's statement is `/var/lib/keel/vip/old-primary-gone`. The operator's stays under `/var/lib/keel/database`. `keel database follow` reads and removes both. - A statement that cannot be written is said, and the claim is recorded with its lease all the same. - `take()` of this node's own claim keeps the lease that is signed into it. - The netns harness runs the VIP helper with the write paths of keel-vip.service, not the whole root of the node. That difference is why CI missed it. ## Tests RED first in `tests/test_mesh_vipetcd.py`: the note is in the state directory; a statement that cannot be written does not stop the claim (the VIP is carried, the lease recorded); a node's own claim taken from the counter keeps its lease and is carried.

navigator added 1 commit 2026-10-15 15:23:15 -03:00

fix: the statement of an expired lease lives in the VIP's state directory (keel#178) ... dd1cdccbb0

On real nodes, the automatic failover of the PostgreSQL pair with etcd
left the VIP on no node. The root half of keel-vip.service wrote the
statement of keel#165 to /var/lib/keel/database/old-primary-gone. The
unit has ProtectSystem=strict and ReadWritePaths=/var/lib/keel/vip
/etc/wireguard, so the write failed with a read-only file system, and
Ops.hold raised before it recorded the claim. The next turn took the
claim from the counter with take(), which kept no lease for this node's
own claim, and the controller carries only a claim with its lease.

- The controller's statement is /var/lib/keel/vip/old-primary-gone. The
  operator's stays /var/lib/keel/database/old-primary-gone. keel
  database follow reads and removes both.
- A statement that cannot be written is said, and the claim is recorded
  with its lease all the same.
- take() of this node's own claim keeps the lease that is signed into
  it, so a claim that etcd holds is renewed and carried.
- The netns harness runs the VIP helper with the write paths of
  keel-vip.service, not the whole root of the node. CI missed the
  failure because of that.

navigator force-pushed fix/vip-gone-note-in-state-dir-178 from dd1cdccbb0 to ce7073eee2 2026-10-15 16:05:35 -03:00 Compare

navigator force-pushed fix/vip-gone-note-in-state-dir-178 from ce7073eee2 to e7205daf77 2026-10-15 16:37:01 -03:00 Compare

navigator merged commit b717c616a4 into main 2026-10-15 17:01:38 -03:00

navigator referenced this pull request from a commit 2026-10-15 17:01:38 -03:00

Merge pull request 'fix: the statement of an expired lease lives in the VIP's state directory (keel#178)' (#179) from fix/vip-gone-note-in-state-dir-178 into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

accessibility

Barrier affecting people with disabilities

  

bug

Something isn't working

  

documentation

Improvements or additions to documentation

  

duplicate

This issue or pull request already exists

  

enhancement

New feature or request

  

good first issue

Good for newcomers

  

help wanted

Extra attention is needed

  

invalid

This doesn't seem right

  

question

Further information is requested

  

wontfix

This will not be worked on

No labels

accessibility

bug

documentation

duplicate

enhancement

good first issue

help wanted

invalid

question

wontfix

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

Keel-Linux/keel!179

No description provided.