fix: a restart of the VIP's unit renews the lease once before it goes (keel#87) #145

Open

marcos-mendez wants to merge 1 commit from fix/vip-restart-renewal into main

marcos-mendez commented

2026-10-10 16:32:26 -03:00

(Migrated from github.com)

keel#87 promised that the VIP survives an upgrade. vip-upgrade / trixie failed on #143 and on main: after a try-restart of keel-vip.service on the holder, a sample found no node carrying the VIP, 4.47 s after the restart (dropped_s: 4.47, holders_seen: [[], ['A']]).

Cause

An address kept across a restart keeps the lifetime of the old controller's last confirmed renewal: RELEASE_AFTER (10 s), less the renewal's age, less 1 s of slack. The next controller extends it only after it has renewed the lease itself.

On the test's link (250 ms ±25 ms, 2% loss), each renewal is a keep-alive plus a linearizable read, each over a new TLS connection. So the last renewal is often seconds old when the restart comes. The stop (1.19 s in the failed run), the start, and the next controller's first renewal took longer than the time left, and the kernel removed the address until the new controller carried it again. This is a product gap, not a test artifact: the sampler already reads each false result a second time (keel#94).

Fix

keel vip tend --stopped (the unit's ExecStopPost, keel/mesh/vipetcd.py: stopped, handed_over) now renews the lease once while the unit restarts, as the controller does. It then gives the address the lifetime that this renewal leaves, so the next controller has the whole release time. The safety bound stays: the lifetime comes from a renewal that the majority confirmed. With no answer from etcd, the address stays as it was and the kernel ends it on time. A lease that is gone drops it. A plain stop renews nothing. docs/vip.md says so. Version 0.23.18 (#144 holds 0.23.17).

Tests

tests/test_mesh_vipupgrade.py (TestHandOver). A restart with a renewal 5 s old renews once (keep-alive, then read) and gives the address the full lifetime. No answer leaves the lifetime as it is. A lease that is gone drops the address. A stop renews nothing. An address not carried again is said. The default client is this member's etcd. These tests failed before the fix (stopped() had no renewal).

Proof on the platform: vip-upgrade / trixie must pass 5 times on this head. I record each run below.

keel#87 promised that the VIP survives an upgrade. `vip-upgrade / trixie` failed on #143 and on main: after a try-restart of keel-vip.service on the holder, a sample found no node carrying the VIP, 4.47 s after the restart (`dropped_s: 4.47`, `holders_seen: [[], ['A']]`). ## Cause An address kept across a restart keeps the lifetime of the old controller's last confirmed renewal: RELEASE_AFTER (10 s), less the renewal's age, less 1 s of slack. The next controller extends it only after it has renewed the lease itself. On the test's link (250 ms ±25 ms, 2% loss), each renewal is a keep-alive plus a linearizable read, each over a new TLS connection. So the last renewal is often seconds old when the restart comes. The stop (1.19 s in the failed run), the start, and the next controller's first renewal took longer than the time left, and the kernel removed the address until the new controller carried it again. This is a product gap, not a test artifact: the sampler already reads each false result a second time (keel#94). ## Fix `keel vip tend --stopped` (the unit's ExecStopPost, `keel/mesh/vipetcd.py`: `stopped`, `handed_over`) now renews the lease once while the unit restarts, as the controller does. It then gives the address the lifetime that this renewal leaves, so the next controller has the whole release time. The safety bound stays: the lifetime comes from a renewal that the majority confirmed. With no answer from etcd, the address stays as it was and the kernel ends it on time. A lease that is gone drops it. A plain stop renews nothing. `docs/vip.md` says so. Version 0.23.18 (#144 holds 0.23.17). ## Tests `tests/test_mesh_vipupgrade.py` (`TestHandOver`). A restart with a renewal 5 s old renews once (keep-alive, then read) and gives the address the full lifetime. No answer leaves the lifetime as it is. A lease that is gone drops the address. A stop renews nothing. An address not carried again is said. The default client is this member's etcd. These tests failed before the fix (`stopped()` had no renewal). Proof on the platform: `vip-upgrade / trixie` must pass 5 times on this head. I record each run below.

This pull request has changes conflicting with the target branch.