etcd-auth / trixie: two VIP holders for 0.2 s while the holder was cut off, on a loaded runner #126

New issue

Closed

opened 2026-10-09 03:34:52 -03:00 by marcos-mendez · 0 comments

marcos-mendez commented

2026-10-09 03:34:52 -03:00

(Migrated from github.com)

When: step 6 of tests/etcd_auth_netns.py, A (the holder and the root's holder) cut off for the renewal test (renew_c_holder_cut, 10.1 s), then back (renew_c_holder_back).

The runner was loaded: three pull requests ran their netns jobs at the same time; another job of the same run measured a maximum RTT of 2.1 s on links that are 250 ms.

Expected (decision 0049): the holder's address has a valid_lft that ends RELEASE_AFTER (10 s) after its last confirmed renewal, and the other member claims only after the lease's TTL (20 s) plus GRACE. A gap of about 10 s should separate the two.

To find out: whether the kernel removed A's address late (addrconf's expiry timer on a busy workqueue; EXPIRY_SLACK is 1 s), whether A's controller carried the address again after the heal before it learned that its lease was gone, or whether B claimed early. The agents' logs in that run's RESULT line do not show the VIP controller's lines for that interval.

This is not caused by #121, #122 or #124 (they do not change the controller, the lease or the address lifetime), but it is a safety signal and must not be hidden by reruns.

Seen in CI run 37892870474 (job etcd-auth / trixie, PR #124, which changes only tests). The sampler saw the VIP on the wg0 of A and of B in two samples, 0.2 s apart (t = 1791527330.62 and 1791527330.83). When: step 6 of tests/etcd_auth_netns.py, A (the holder and the root's holder) cut off for the renewal test (`renew_c_holder_cut`, 10.1 s), then back (`renew_c_holder_back`). The runner was loaded: three pull requests ran their netns jobs at the same time; another job of the same run measured a maximum RTT of 2.1 s on links that are 250 ms. Expected (decision 0049): the holder's address has a valid_lft that ends RELEASE_AFTER (10 s) after its last confirmed renewal, and the other member claims only after the lease's TTL (20 s) plus GRACE. A gap of about 10 s should separate the two. To find out: whether the kernel removed A's address late (addrconf's expiry timer on a busy workqueue; EXPIRY_SLACK is 1 s), whether A's controller carried the address again after the heal before it learned that its lease was gone, or whether B claimed early. The agents' logs in that run's RESULT line do not show the VIP controller's lines for that interval. This is not caused by #121, #122 or #124 (they do not change the controller, the lease or the address lifetime), but it is a safety signal and must not be hidden by reruns.