fix: upgrades keep the VIP; etcd restarts one member at a time #46

Merged

marcos-mendez merged 6 commits from feat/vip-etcd-rolling-upgrades into 19.x

2026-10-08 04:56:13 -03:00

marcos-mendez commented

2026-10-08 03:38:23 -03:00

(Migrated from github.com)

The packaging half of Keel-Linux/keel#87, for the maintainer's requirement of 2026-10-10: an apt full-upgrade on a node whose VIP is active must neither drop the VIP nor trigger a failover. The design, and the measurements, are in keel#87.

What an upgrade did before this PR

keel-overlay-vip 0.1.1. dh_installsystemd --no-enable --no-start generated a preinst that runs deb-systemd-invoke stop keel-vip-check.timer keel-vip.service on every upgrade, and a postinst that never starts either unit again. On the holder, ExecStopPost dropped the VIP. On the replica, the controller was stopped as well, so nothing claimed the VIP. The VIP was down until a reboot or keel spec apply.
keel-overlay-etcd 0.2.0. No upgrade action of its own. etcd-server's postinst restarts etcd.service on every upgrade, and nothing coordinates that restart across members. With apt running on two members of three at once, the cluster loses its majority while both restart.

Changes

keel-overlay-vip 0.2.0
- Built with --no-stop-on-upgrade, so there is no preinst stop.
- The postinst runs deb-systemd-invoke try-restart keel-vip.service on an upgrade, and on a file trigger for /usr/lib/python3/dist-packages/keel, so an upgrade of keel restarts the helper and it runs the new code. A unit that was stopped is never started.
- RestartSec=2.
- Depends on keel (>= 0.21.0). That version keeps the address across the restart: the kernel bounds it with a lifetime, ExecStopPost keeps it while the unit restarts, and the next controller renews the same lease.
keel-overlay-etcd 0.3.0
- Ships /usr/lib/systemd/system/etcd.service.d/keel-overlay-etcd.conf:
  - ExecStop=+keel mesh etcd gate stop waits until every other voter is healthy and no other member is restarting, then takes a restart lock held on an etcd lease.
  - ExecStartPost=-+keel mesh etcd gate started releases the lock once this member is back in the majority.
- The stop is never refused for good: the gate gives up after 300 s, and at shutdown it does not wait.
- A node with no etcd cluster passes the gate at once, and so does a member that does not answer (a crash).
- The postinst and postrm run daemon-reload.
- Depends on keel (>= 0.21.0).
tests/overlay-install.bats: real upgrades to builds of the same packages with a bumped version (with_version), with fab's policy-rc.d moved aside for those tests. The cases:
- a running controller is try-restarted, with a new invocation, and its timer runs;
- a stopped controller stays stopped;
- an upgrade of keel restarts the controller through the trigger;
- etcd restarts through the gate in well under the gate's wait.
The install job builds keel at the head of keel#87. Pin it to keel#87's merge commit once that PR is merged. keel#87 should merge first.

Test plan

build / trixie: both packages build and pass lintian (clean locally).
install / trixie: overlay-install.bats, including the four new upgrade tests.
After keel#87 merges, re-pin KEEL_COMMIT to its merge commit.

The packaging half of Keel-Linux/keel#87, for the maintainer's requirement of 2026-10-10: an `apt full-upgrade` on a node whose VIP is active must neither drop the VIP nor trigger a failover. The design, and the measurements, are in keel#87. ## What an upgrade did before this PR - **keel-overlay-vip 0.1.1.** `dh_installsystemd --no-enable --no-start` generated a preinst that runs `deb-systemd-invoke stop keel-vip-check.timer keel-vip.service` on every upgrade, and a postinst that never starts either unit again. On the holder, ExecStopPost dropped the VIP. On the replica, the controller was stopped as well, so nothing claimed the VIP. **The VIP was down until a reboot or `keel spec apply`.** - **keel-overlay-etcd 0.2.0.** No upgrade action of its own. etcd-server's postinst restarts `etcd.service` on every upgrade, and nothing coordinates that restart across members. With apt running on two members of three at once, the cluster loses its majority while both restart. ## Changes - **keel-overlay-vip 0.2.0** - Built with `--no-stop-on-upgrade`, so there is no preinst stop. - The postinst runs `deb-systemd-invoke try-restart keel-vip.service` on an upgrade, and on a file trigger for `/usr/lib/python3/dist-packages/keel`, so an upgrade of keel restarts the helper and it runs the new code. A unit that was stopped is never started. - `RestartSec=2`. - Depends on keel (>= 0.21.0). That version keeps the address across the restart: the kernel bounds it with a lifetime, ExecStopPost keeps it while the unit restarts, and the next controller renews the same lease. - **keel-overlay-etcd 0.3.0** - Ships `/usr/lib/systemd/system/etcd.service.d/keel-overlay-etcd.conf`: - `ExecStop=+keel mesh etcd gate stop` waits until every other voter is healthy and no other member is restarting, then takes a restart lock held on an etcd lease. - `ExecStartPost=-+keel mesh etcd gate started` releases the lock once this member is back in the majority. - The stop is never refused for good: the gate gives up after 300 s, and at shutdown it does not wait. - A node with no etcd cluster passes the gate at once, and so does a member that does not answer (a crash). - The postinst and postrm run `daemon-reload`. - Depends on keel (>= 0.21.0). - **`tests/overlay-install.bats`**: real upgrades to builds of the same packages with a bumped version (`with_version`), with fab's policy-rc.d moved aside for those tests. The cases: - a running controller is try-restarted, with a new invocation, and its timer runs; - a stopped controller stays stopped; - an upgrade of keel restarts the controller through the trigger; - etcd restarts through the gate in well under the gate's wait. - The install job builds keel at the head of keel#87. **Pin it to keel#87's merge commit once that PR is merged.** keel#87 should merge first. ## Test plan - [ ] `build / trixie`: both packages build and pass lintian (clean locally). - [ ] `install / trixie`: overlay-install.bats, including the four new upgrade tests. - [ ] After keel#87 merges, re-pin `KEEL_COMMIT` to its merge commit.