vip: a claim made before etcd stays without a lease after etcd forms, so no lease fence and no automatic failover #161

New issue

Open

opened 2026-10-15 08:00:15 -03:00 by navigator · 0 comments

navigator commented

2026-10-15 08:00:15 -03:00

Owner

Found on real nodes on 2026-10-15 (keel 0.23.21, handbook decision 0057, evidence docs/evidence/2026-10-15-br3-thin-etcd-witness.md).

What happened

The PostgreSQL pair 9062 (BR2) and 9054 (BR1) was cloud_simple, with the VIP fd1a:bb4c:b407::ffff:1 at epoch 8. The claim came from the members' channel, so it has no lease. A witness joined, both pair nodes moved to cloud_advanced by their specs, and keel mesh etcd form formed a cluster of 3. keel-vip.service started on both pair nodes.

After the form, and after one run of keel mesh etcd tend:

etcd holds no key under /keel/<mesh>/vip/. The counter is empty.
/var/lib/keel/vip/<vip>.json on the holder says "lease": null.
The address on the holder has valid_lft forever. The controller does not renew a lease, and the kernel never removes the address.
keel vip promote on the holder prints "this node holds ... already, at epoch 8" and exits 0 (promoted_etcd).
keel vip status shows the same output as before etcd.

Why

Controller.own() takes only claims with a lease, so holding() does nothing for the claim of epoch 8. fail_over() on the other node returns when held.claim.lease is None. Thus the pair has etcd, but it has no lease fence and no automatic failover. Nothing says so.

Expected

When etcd forms, or at the first turn of the controller, the holder of a claim without a lease claims again by the counter's transaction at the next epoch. Or keel vip status and keel mesh status say that the claim has no lease, and name the command that moves it.

Workaround

A planned keel database promote (no --old-primary-gone) on the replica. It claims at the next epoch by the counter's transaction, with a lease.

Found on real nodes on 2026-10-15 (keel 0.23.21, handbook decision 0057, evidence `docs/evidence/2026-10-15-br3-thin-etcd-witness.md`). ## What happened The PostgreSQL pair 9062 (BR2) and 9054 (BR1) was `cloud_simple`, with the VIP `fd1a:bb4c:b407::ffff:1` at epoch 8. The claim came from the members' channel, so it has no lease. A witness joined, both pair nodes moved to `cloud_advanced` by their specs, and `keel mesh etcd form` formed a cluster of 3. `keel-vip.service` started on both pair nodes. After the form, and after one run of `keel mesh etcd tend`: - etcd holds no key under `/keel/<mesh>/vip/`. The counter is empty. - `/var/lib/keel/vip/<vip>.json` on the holder says `"lease": null`. - The address on the holder has `valid_lft forever`. The controller does not renew a lease, and the kernel never removes the address. - `keel vip promote` on the holder prints "this node holds ... already, at epoch 8" and exits 0 (`promoted_etcd`). - `keel vip status` shows the same output as before etcd. ## Why `Controller.own()` takes only claims with a lease, so `holding()` does nothing for the claim of epoch 8. `fail_over()` on the other node returns when `held.claim.lease is None`. Thus the pair has etcd, but it has no lease fence and no automatic failover. Nothing says so. ## Expected When etcd forms, or at the first turn of the controller, the holder of a claim without a lease claims again by the counter's transaction at the next epoch. Or `keel vip status` and `keel mesh status` say that the claim has no lease, and name the command that moves it. ## Workaround A planned `keel database promote` (no `--old-primary-gone`) on the replica. It claims at the next epoch by the counter's transaction, with a lease.