database: with etcd, keel-database-follow.path triggers itself every 1 to 2 s and fails with unit-start-limit-hit #164

New issue

Closed

opened 2026-10-15 08:21:06 -03:00 by navigator · 0 comments

navigator commented

2026-10-15 08:21:06 -03:00

Owner

Found on real nodes on 2026-10-15 (keel 0.23.21, etcd of 3 members in 3 sites, handbook decision 0057, evidence docs/evidence/2026-10-15-br3-thin-etcd-witness.md).

What happened

On both nodes of the PostgreSQL pair, keel-database-follow.path (PathChanged=/var/lib/keel/vip) starts keel-database-follow.service again 0.2 to 1 s after each run ends. The VIP did not move.

On the holder (9062), the service ran every 2.2 s on average, from 1792062818.80 to the end of the record. Each run takes 1.0 to 1.9 s.
On the replica (9054), 5 runs came in 7 s. Then: keel-database-follow.service: Start request repeated too quickly and keel-database-follow.path: Failed with result 'unit-start-limit-hit' (1792062823.43).
The same happened earlier on both nodes, in a diverged state (each run exits 16): the path failed at 1792062087.20 on 9062.

After the failure, nothing starts the path again. keel spec apply --system-only does not start it, and keel diff exits 0 (no drift). A VIP move is then seen only by keel-database-watch.timer (30 s).

The restore was by hand: systemctl reset-failed keel-database-follow.path keel-database-follow.service; systemctl start keel-database-follow.path. The path failed again on the replica 8 s later.

Probable cause

Something in /var/lib/keel/vip changes on each run or on each lease renewal (every 2 s with etcd), so the path sees a change that is not a VIP move. Before etcd (2026-10-15 00:10 UTC, no controller) the path stayed active.

Expected

The path triggers only when the claim changes. A run that changes nothing does not trigger the next one. keel diff reports a failed unit of an enabled overlay as drift, and apply --system starts it.

Found on real nodes on 2026-10-15 (keel 0.23.21, etcd of 3 members in 3 sites, handbook decision 0057, evidence `docs/evidence/2026-10-15-br3-thin-etcd-witness.md`). ## What happened On both nodes of the PostgreSQL pair, `keel-database-follow.path` (`PathChanged=/var/lib/keel/vip`) starts `keel-database-follow.service` again 0.2 to 1 s after each run ends. The VIP did not move. - On the holder (9062), the service ran every 2.2 s on average, from 1792062818.80 to the end of the record. Each run takes 1.0 to 1.9 s. - On the replica (9054), 5 runs came in 7 s. Then: `keel-database-follow.service: Start request repeated too quickly` and `keel-database-follow.path: Failed with result 'unit-start-limit-hit'` (1792062823.43). - The same happened earlier on both nodes, in a diverged state (each run exits 16): the path failed at 1792062087.20 on 9062. After the failure, nothing starts the path again. `keel spec apply --system-only` does not start it, and `keel diff` exits 0 (no drift). A VIP move is then seen only by `keel-database-watch.timer` (30 s). The restore was by hand: `systemctl reset-failed keel-database-follow.path keel-database-follow.service; systemctl start keel-database-follow.path`. The path failed again on the replica 8 s later. ## Probable cause Something in `/var/lib/keel/vip` changes on each run or on each lease renewal (every 2 s with etcd), so the path sees a change that is not a VIP move. Before etcd (2026-10-15 00:10 UTC, no controller) the path stayed active. ## Expected The path triggers only when the claim changes. A run that changes nothing does not trigger the next one. `keel diff` reports a failed unit of an enabled overlay as drift, and `apply --system` starts it.