fix: with etcd, the other node claims the VIP only when the lease shows -1 (keel#126) #127

Merged

marcos-mendez merged 1 commit from fix/vip-lease-last-second into main

2026-10-09 05:03:38 -03:00

marcos-mendez commented

2026-10-09 04:13:01 -03:00

(Migrated from github.com)

Closes #126.

Cause

This is a real overlap in the product, not an artifact of the sampler.

etcd gives a lease's time to live in whole seconds, rounded down. A lease with less than 1 s left shows ttl 0, and a keep-alive in that second keeps it (seen on etcd 3.5.16: ttl 0, then keepalived with TTL(3), then ttl 2). Controller.fail_over and vippromote.gone_lease took ttl <= 0 as a lease that is gone.

In run 37892870474 (etcd-auth / trixie), A was cut off for 10.1 s from about t = ...312.6 to ...322.8. Its lease expired about 20 s after its last keep-alive before the cut, at about ...330.5. After the heal, A's renewals failed for some seconds more (loaded runner, round trips to 2.1 s). Its first confirmed renewal came in the last second of the lease. B read 0 in that second and claimed. The sampler read both A and B as holders at t = 1791527330.62 and 1791527330.83, within the same millisecond each time. A dropped the VIP when it took B's newer claim from the counter (1 s turn). The fake etcd of the unit tests gave a live lease 1 s at least (max(1, ...)), so this case never ran.

Fix

vipetcd.gone(ttl): only -1 is gone. The failover and promote use it.
The fake etcd rounds down as etcd does. New tests: a lease in its last second, renewed by the holder, is not taken; the replica claims at -1, within 1.5 s of the expiry; promote waits for -1.
Margin, in docs/vip.md: the old holder's address is gone at s + 10 s (by self-release and by valid_lft); a claim is possible at s + 20 s at the earliest; 10 s or more, about 10.2 s on the design case. Delay and loss can only make the margin larger.

Netns assertion

tests/vip_overlap.py is a process of its own. It reads every node's /proc/PID/net/if_inet6 in a tight loop (2 ms), in a palindromic order (A B C C B A, the first node turned each round), with CLOCK_MONOTONIC. It reports an overlap only when it can prove one at one instant, and a handover is never one. vip / trixie and etcd-auth / trixie fail on any such overlap. The 200 ms sampler stays.

Version 0.23.9.

Closes #126. ## Cause This is a real overlap in the product, not an artifact of the sampler. etcd gives a lease's time to live in whole seconds, rounded down. A lease with less than 1 s left shows `ttl 0`, and a keep-alive in that second keeps it (seen on etcd 3.5.16: `ttl 0`, then `keepalived with TTL(3)`, then `ttl 2`). `Controller.fail_over` and `vippromote.gone_lease` took `ttl <= 0` as a lease that is gone. In run 37892870474 (`etcd-auth / trixie`), A was cut off for 10.1 s from about t = ...312.6 to ...322.8. Its lease expired about 20 s after its last keep-alive before the cut, at about ...330.5. After the heal, A's renewals failed for some seconds more (loaded runner, round trips to 2.1 s). Its first confirmed renewal came in the last second of the lease. B read 0 in that second and claimed. The sampler read both A and B as holders at t = 1791527330.62 and 1791527330.83, within the same millisecond each time. A dropped the VIP when it took B's newer claim from the counter (1 s turn). The fake etcd of the unit tests gave a live lease 1 s at least (`max(1, ...)`), so this case never ran. ## Fix - `vipetcd.gone(ttl)`: only -1 is gone. The failover and promote use it. - The fake etcd rounds down as etcd does. New tests: a lease in its last second, renewed by the holder, is not taken; the replica claims at -1, within 1.5 s of the expiry; promote waits for -1. - Margin, in docs/vip.md: the old holder's address is gone at s + 10 s (by self-release and by valid_lft); a claim is possible at s + 20 s at the earliest; 10 s or more, about 10.2 s on the design case. Delay and loss can only make the margin larger. ## Netns assertion `tests/vip_overlap.py` is a process of its own. It reads every node's `/proc/PID/net/if_inet6` in a tight loop (2 ms), in a palindromic order (A B C C B A, the first node turned each round), with CLOCK_MONOTONIC. It reports an overlap only when it can prove one at one instant, and a handover is never one. `vip / trixie` and `etcd-auth / trixie` fail on any such overlap. The 200 ms sampler stays. Version 0.23.9.