feat: upgrades keep the VIP: kernel-bounded address, graceful restart, rolling etcd #87

Merged

marcos-mendez merged 7 commits from feat/vip-continuity-upgrades into main

2026-10-08 04:30:11 -03:00

marcos-mendez commented

2026-10-08 03:08:19 -03:00

(Migrated from github.com)

The maintainer's requirement of 2026-10-10: an apt full-upgrade on the node that holds the VIP must not drop the VIP or trigger a failover, beyond a short planned blip that is measured and documented. The same applies to the replica and to every etcd member. The units are in Keel-Linux/common (keel-overlay-vip, keel-overlay-etcd). That PR is the packaging half of this one.

What an upgrade does today (`befaaea`, keel-overlay-vip 0.1.1)

keel-overlay-vip. dh_installsystemd --no-enable --no-start gives a preinst that runs deb-systemd-invoke stop keel-vip-check.timer keel-vip.service on upgrade, and a postinst that never starts them again. On every node, an upgrade of the package stops the helper, its controller and the check, and leaves them stopped. --stopped (ExecStopPost) and the helper's own finally remove the address from wg0, so the holder drops the VIP. The replica's controller is stopped too, so nothing claims it. This is an outage until a reboot or keel spec apply, not a blip.
keel. Its postinst restarts keel-mesh-members, keel-mesh-sync and keel-mesh-etcd (restart-after-upgrade). The preinst stops keel-firewall and keel-network-revert (--no-start) without starting them again. Both are oneshots with no ExecStop, so the nft table stays loaded. Nothing restarts keel-vip.service, so the helper keeps running the old code. Its lazy imports (vipmsg in held_of, memberlink) can then load files from the new version into the old process.
keel-overlay-etcd. It owns no unit and has no upgrade action of its own.
etcd-server. Its postinst runs deb-systemd-invoke restart etcd.service on every upgrade. Nothing coordinates this across nodes: run apt on two members at once and quorum is lost for the length of both restarts. On the holder, the controller asks only its local member ([::1]:2379), so the local restart stops renewals. If the restart outlasts the 10 s release window, the VIP blips. If it outlasts the 20 s TTL, there is a failover and the holder is fenced.
A restart of keel-vip.service on the holder (by hand, or by a fixed postinst): SIGTERM reaches serve's finally, and vipetcd.stopped() removes the address, as --stopped does again. The new controller starts with nothing in memory and carries the address only after its first renewal confirmed by the majority (keepalive plus a linearizable read). The VIP is therefore down for the whole stop, start and first renewal. The lease (TTL 20 s) survives that, so there is no failover unless the restart takes longer than the TTL. The measured values are in "Measured" below.

Design

The kernel enforces the release deadline. The holder carries the address with a finite valid_lft: ip -6 addr replace <vip>/128 dev wg0 valid_lft N preferred_lft 0 nodad, with N = floor(RELEASE_AFTER - age) - 1. Here age is the time since the renewal the majority confirmed (CLOCK_BOOTTIME), and the 1 s covers the kernel's lateness: addrconf runs its check at most 0.25 s after an address expires. Each confirmed renewal (every 2 s) replaces the address with a new lifetime. If nothing renews it, the kernel removes the address before 10 s after the last confirmed renewal. That is the same deadline the controller enforced from its memory. It now holds even when no process enforces it: a controller stopped, restarting, crashed or frozen. That last case, the "one case this does not cover" in docs/vip.md, is now covered.
Graceful controller restart. The helper stopping no longer removes a bounded address. A new controller that finds a bounded address on wg0 leaves it alone. It does not extend the address either: it renews the same lease, recorded in the state file with the claim, at once, and replaces the lifetime only once the majority confirms that renewal. If the lease is gone (expired or revoked), it drops the address and fences, or claims again, exactly as today. The split-brain rules do not change. The address is never on wg0 later than RELEASE_AFTER after a renewal the majority confirmed, and only the kernel can carry it past a process's death. A reboot always starts with no address, and with the same rule as today (nothing carried until a renewal). An address with no bound (valid_lft forever, left by an older keel) is dropped at start and at stop, as today.
ExecStopPost decides. keel vip tend --stopped drops every VIP unless the unit is restarting. That covers a restart or try-restart job queued for the unit (systemd's job list), or a helper that died ($SERVICE_RESULT other than success, which Restart=always restarts). In both cases the bounded address stays, and the kernel removes it if no controller renews it. A plain systemctl stop, keel spec apply disabling the overlay, or a shutdown still drops the VIP at once.
The controller renews through any member. The helper gives the controller the loopback endpoint first, then every other member's client URL from the cluster record. A renewal still counts only once a linearizable read answers after it, so the majority confirms it whichever member relays it. A cut-off holder reaches no other member either, so nothing about the cut-off case changes. The holder's own etcd restarting no longer stops renewals. A local member that does not answer (a crash) passes the gate at once.
etcd restarts one member at a time. keel mesh etcd gate stop runs as ExecStop of etcd.service, from keel-overlay-etcd's drop-in. Before etcd is sent SIGTERM, it waits until the other voters are healthy enough to keep quorum without this member. It also waits until no other member holds the restart lock, then takes it. The lock is /keel/<mesh>/etcd/restarting, written by compare-and-swap on an etcd lease (TTL 120 s). keel mesh etcd gate started runs as ExecStartPost. It waits until the local member answers a linearizable read, then deletes its own lock. A stop cannot be refused for good, so after --wait (300 s by default) the gate says so loudly and lets the stop go ahead. At shutdown it does not wait. Apt run on two members at once therefore restarts them one after the other: the second member's postinst blocks in deb-systemd-invoke restart until the first is healthy again. etcd itself hands its leadership to another member on SIGTERM.
keel mesh upgrade-check is read-only and exits 1 when this node should not be upgraded now. It names each unhealthy voter, the member holding the restart lock, and whether this node holds a VIP. An operator, or a fleet tool, runs it before apt full-upgrade.
Packaging is in the common PR. keel-overlay-vip gets --no-stop-on-upgrade, a postinst try-restart keel-vip.service on upgrade, a trigger on keel's Python files (an upgrade of keel try-restarts the helper, so the new code runs), and RestartSec=2. keel-overlay-etcd ships the etcd.service drop-in. keel's own units need no change for the VIP.
Docs. docs/vip.md gets "Upgrading a pair without downtime", and docs/mesh.md gets etcd's rolling restart. The procedure is one node at a time, the holder last or after a planned keel vip promote on the replica (about 3 to 4 s of planned downtime, measured by the vip job's (b)), and keel mesh upgrade-check before each node.

The first upgrade into this version still blips once on the holder. The old helper's in-memory finally drops the address before the new code runs, and the new controller carries it again after its first renewal.

Tests

Unit tests for each piece above: the lifetime and its arithmetic, a restarted controller that keeps a bounded address and drops an unbounded one, refreshing on every renewal, --stopped on a restart, a crash and a stop, the fallback endpoints, and the gate and upgrade-check against a fake etcd.
tests/test_vip_upgrade_netns.py (CI job vip-upgrade / trixie) runs on the vip job's nodes and links: 250 ms ±25 ms and 2% loss, measured first. Scenarios: (a) try-restart of keel-vip.service on the holder (three times) and on the replica, and a SIGKILL of the holder's helper; (b) etcd restarted through the gate on C, then on the holder A, then on B; (c) C held down, B's gate refuses for its wait while upgrade-check names C, then lets B through once C is back; (d) the holder's controller frozen with SIGSTOP, so the kernel drops the address within the release window and B claims after the lease. The checks: no failover (the counter's epoch does not move), never two holders, and a ping gap from C under 1 s for each upgrade-style restart.
A real upgrade between built .debs, version-bumped in CI, is in common's install / trixie (tests/overlay-install.bats).

Measured (vip-upgrade / trixie, 250 ms ±25 ms with 2% loss)

The longest gap in C's answers to the VIP, which it pings every 50 ms:

Scenario	`befaaea` (red run)	This PR (three green runs)
try-restart of keel-vip.service on the holder (×3)	2.75, 2.99, 3.17 s; the address dropped	0.12 to 0.30 s; the address never left wg0
try-restart on the replica	0.16 s	0.13 to 0.28 s
SIGKILL of the holder's helper	4.40 s	0.14 to 0.17 s
etcd restarted on C, then A (the holder), then B	0.15 / 0.18 / 0.14 s, with no gate	0.13 to 0.39 s; 10 to 15 s per restart, gate included
the holder's controller frozen	A and B both carried the VIP for 0.8 s (A kept it until B's claim reached it)	the kernel dropped the address 6.2 to 6.9 s after the freeze; B carried it at 21 s; never two holders

There was no failover in any upgrade-style restart: the counter's epoch did not move.
In every run, the other two members kept writing while one member's etcd restarted (quorum held).
(c) B's gate waited its 20 s and gave up with exit 16 while C was down, and B's etcd kept running. upgrade-check on A named C, both as unhealthy and as holding the restart lock. Once C was back, B's gate passed in 5 to 7 s.
Other jobs in this PR:
- The vip / trixie hardening check read the controller's capabilities from systemd's executor, before it exec'd the controller (CapEff 0x200100). It now waits for the exec. That race is what failed main's run of #88.
- etcd / trixie failed once on a learner add ("unhealthy cluster"). The test does not touch this code, and it passed on a rerun.

Common PR: Keel-Linux/common#46. Its install / trixie job builds this branch's keel. Merge this PR first, then re-pin common#46 to the merge commit.

The maintainer's requirement of 2026-10-10: an `apt full-upgrade` on the node that holds the VIP must not drop the VIP or trigger a failover, beyond a short planned blip that is measured and documented. The same applies to the replica and to every etcd member. The units are in Keel-Linux/common (keel-overlay-vip, keel-overlay-etcd). That PR is the packaging half of this one. ## What an upgrade does today (befaaea, keel-overlay-vip 0.1.1) - **keel-overlay-vip.** `dh_installsystemd --no-enable --no-start` gives a preinst that runs `deb-systemd-invoke stop keel-vip-check.timer keel-vip.service` on upgrade, and a postinst that never starts them again. On every node, an upgrade of the package stops the helper, its controller and the check, and leaves them stopped. `--stopped` (ExecStopPost) and the helper's own `finally` remove the address from wg0, so the holder drops the VIP. The replica's controller is stopped too, so nothing claims it. **This is an outage until a reboot or `keel spec apply`, not a blip.** - **keel.** Its postinst restarts `keel-mesh-members`, `keel-mesh-sync` and `keel-mesh-etcd` (restart-after-upgrade). The preinst stops `keel-firewall` and `keel-network-revert` (`--no-start`) without starting them again. Both are oneshots with no ExecStop, so the nft table stays loaded. Nothing restarts `keel-vip.service`, so the helper keeps running the old code. Its lazy imports (`vipmsg` in `held_of`, `memberlink`) can then load files from the new version into the old process. - **keel-overlay-etcd.** It owns no unit and has no upgrade action of its own. - **etcd-server.** Its postinst runs `deb-systemd-invoke restart etcd.service` on every upgrade. Nothing coordinates this across nodes: run apt on two members at once and quorum is lost for the length of both restarts. On the holder, the controller asks only its local member (`[::1]:2379`), so the local restart stops renewals. If the restart outlasts the 10 s release window, the VIP blips. If it outlasts the 20 s TTL, there is a failover and the holder is fenced. - **A restart of keel-vip.service on the holder** (by hand, or by a fixed postinst): SIGTERM reaches `serve`'s `finally`, and `vipetcd.stopped()` removes the address, as `--stopped` does again. The new controller starts with nothing in memory and carries the address only after its first renewal confirmed by the majority (keepalive plus a linearizable read). The VIP is therefore down for the whole stop, start and first renewal. The lease (TTL 20 s) survives that, so there is no failover unless the restart takes longer than the TTL. The measured values are in "Measured" below. ## Design 1. **The kernel enforces the release deadline.** The holder carries the address with a finite `valid_lft`: `ip -6 addr replace <vip>/128 dev wg0 valid_lft N preferred_lft 0 nodad`, with `N = floor(RELEASE_AFTER - age) - 1`. Here `age` is the time since the renewal the majority confirmed (CLOCK_BOOTTIME), and the 1 s covers the kernel's lateness: addrconf runs its check at most 0.25 s after an address expires. Each confirmed renewal (every 2 s) replaces the address with a new lifetime. If nothing renews it, the kernel removes the address before 10 s after the last confirmed renewal. That is the same deadline the controller enforced from its memory. It now holds even when no process enforces it: a controller stopped, restarting, crashed or frozen. That last case, the "one case this does not cover" in docs/vip.md, is now covered. 2. **Graceful controller restart.** The helper stopping no longer removes a bounded address. A new controller that finds a bounded address on wg0 leaves it alone. It does not extend the address either: it renews the same lease, recorded in the state file with the claim, at once, and replaces the lifetime only once the majority confirms that renewal. If the lease is gone (expired or revoked), it drops the address and fences, or claims again, exactly as today. The split-brain rules do not change. The address is never on wg0 later than RELEASE_AFTER after a renewal the majority confirmed, and only the kernel can carry it past a process's death. A reboot always starts with no address, and with the same rule as today (nothing carried until a renewal). An address with no bound (`valid_lft forever`, left by an older keel) is dropped at start and at stop, as today. 3. **ExecStopPost decides.** `keel vip tend --stopped` drops every VIP **unless the unit is restarting**. That covers a `restart` or `try-restart` job queued for the unit (systemd's job list), or a helper that died (`$SERVICE_RESULT` other than `success`, which Restart=always restarts). In both cases the bounded address stays, and the kernel removes it if no controller renews it. A plain `systemctl stop`, `keel spec apply` disabling the overlay, or a shutdown still drops the VIP at once. 4. **The controller renews through any member.** The helper gives the controller the loopback endpoint first, then every other member's client URL from the cluster record. A renewal still counts only once a linearizable read answers after it, so the majority confirms it whichever member relays it. A cut-off holder reaches no other member either, so nothing about the cut-off case changes. The holder's own etcd restarting no longer stops renewals. A local member that does not answer (a crash) passes the gate at once. 5. **etcd restarts one member at a time.** `keel mesh etcd gate stop` runs as ExecStop of etcd.service, from keel-overlay-etcd's drop-in. Before etcd is sent SIGTERM, it waits until the other voters are healthy enough to keep quorum without this member. It also waits until no other member holds the restart lock, then takes it. The lock is `/keel/<mesh>/etcd/restarting`, written by compare-and-swap on an etcd lease (TTL 120 s). `keel mesh etcd gate started` runs as ExecStartPost. It waits until the local member answers a linearizable read, then deletes its own lock. A stop cannot be refused for good, so after `--wait` (300 s by default) the gate says so loudly and lets the stop go ahead. At shutdown it does not wait. Apt run on two members at once therefore restarts them one after the other: the second member's postinst blocks in `deb-systemd-invoke restart` until the first is healthy again. etcd itself hands its leadership to another member on SIGTERM. 6. **`keel mesh upgrade-check`** is read-only and exits 1 when this node should not be upgraded now. It names each unhealthy voter, the member holding the restart lock, and whether this node holds a VIP. An operator, or a fleet tool, runs it before `apt full-upgrade`. 7. **Packaging** is in the common PR. keel-overlay-vip gets `--no-stop-on-upgrade`, a postinst `try-restart keel-vip.service` on upgrade, a trigger on keel's Python files (an upgrade of keel try-restarts the helper, so the new code runs), and `RestartSec=2`. keel-overlay-etcd ships the etcd.service drop-in. keel's own units need no change for the VIP. 8. **Docs.** docs/vip.md gets "Upgrading a pair without downtime", and docs/mesh.md gets etcd's rolling restart. The procedure is one node at a time, the holder last or after a planned `keel vip promote` on the replica (about 3 to 4 s of planned downtime, measured by the vip job's (b)), and `keel mesh upgrade-check` before each node. The first upgrade *into* this version still blips once on the holder. The old helper's in-memory `finally` drops the address before the new code runs, and the new controller carries it again after its first renewal. ## Tests - Unit tests for each piece above: the lifetime and its arithmetic, a restarted controller that keeps a bounded address and drops an unbounded one, refreshing on every renewal, `--stopped` on a restart, a crash and a stop, the fallback endpoints, and the gate and upgrade-check against a fake etcd. - `tests/test_vip_upgrade_netns.py` (CI job **vip-upgrade / trixie**) runs on the vip job's nodes and links: 250 ms ±25 ms and 2% loss, measured first. Scenarios: (a) try-restart of keel-vip.service on the holder (three times) and on the replica, and a SIGKILL of the holder's helper; (b) etcd restarted through the gate on C, then on the holder A, then on B; (c) C held down, B's gate refuses for its wait while `upgrade-check` names C, then lets B through once C is back; (d) the holder's controller frozen with SIGSTOP, so the kernel drops the address within the release window and B claims after the lease. The checks: no failover (the counter's epoch does not move), never two holders, and a ping gap from C under 1 s for each upgrade-style restart. - A real upgrade between built `.deb`s, version-bumped in CI, is in common's `install / trixie` (tests/overlay-install.bats). ## Measured (vip-upgrade / trixie, 250 ms ±25 ms with 2% loss) The longest gap in C's answers to the VIP, which it pings every 50 ms: | Scenario | befaaea (red run) | This PR (three green runs) | | --- | --- | --- | | try-restart of keel-vip.service on the holder (×3) | 2.75, 2.99, 3.17 s; the address dropped | 0.12 to 0.30 s; the address never left wg0 | | try-restart on the replica | 0.16 s | 0.13 to 0.28 s | | SIGKILL of the holder's helper | 4.40 s | 0.14 to 0.17 s | | etcd restarted on C, then A (the holder), then B | 0.15 / 0.18 / 0.14 s, with no gate | 0.13 to 0.39 s; 10 to 15 s per restart, gate included | | the holder's controller frozen | **A and B both carried the VIP for 0.8 s** (A kept it until B's claim reached it) | the kernel dropped the address 6.2 to 6.9 s after the freeze; B carried it at 21 s; never two holders | - There was no failover in any upgrade-style restart: the counter's epoch did not move. - In every run, the other two members kept writing while one member's etcd restarted (quorum held). - (c) B's gate waited its 20 s and gave up with exit 16 while C was down, and B's etcd kept running. `upgrade-check` on A named C, both as unhealthy and as holding the restart lock. Once C was back, B's gate passed in 5 to 7 s. - Other jobs in this PR: - The **vip / trixie** hardening check read the controller's capabilities from systemd's executor, before it exec'd the controller (CapEff 0x200100). It now waits for the exec. That race is what failed main's run of #88. - **etcd / trixie** failed once on a learner add ("unhealthy cluster"). The test does not touch this code, and it passed on a rerun. Common PR: Keel-Linux/common#46. Its `install / trixie` job builds this branch's keel. Merge this PR first, then re-pin common#46 to the merge commit.