database: with etcd, the writes resume 12 s after the VIP moves on keel database promote --old-primary-gone (bound 5 s, keel#148) #169

New issue

Closed

opened 2026-10-15 08:42:25 -03:00 by navigator · 0 comments

navigator commented

2026-10-15 08:42:25 -03:00

Owner

Found on real nodes on 2026-10-15 (keel 0.23.21, etcd of 3 members in 3 sites: 9054 BR1, 9062 BR2, witness 9071 BR3; handbook decision 0057, evidence docs/evidence/2026-10-15-br3-thin-etcd-witness.md).

Case (c) of docs/replication-postgresql.md, the old primary cut off (WireGuard dropped), then keel database promote --old-primary-gone on the standby:

	VIP on B	B out of recovery	First write acknowledged	From the VIP
base	1792062927.87	2939.45	2939.76	11.9 s
netem 250 ms	1792064189.10	4201.02	4201.99	12.9 s
2026-10-14, no etcd				0.83 s and 1.47 s
CI				1.1 to 1.9 s

In both runs keel-database-follow.path on B was failed (keel#164), so the promotion came from the promote command's own follow step, after it told the peers (1.5 s bound for the old primary, then the witness). Before etcd, the path ran follow as soon as the VIP moved.

The bound of 5 s from the VIP on B is not met. The cause is probably keel#164, but the follow step itself takes about 10 s here, and that is not explained yet. Re-measure after keel#164.

Found on real nodes on 2026-10-15 (keel 0.23.21, etcd of 3 members in 3 sites: 9054 BR1, 9062 BR2, witness 9071 BR3; handbook decision 0057, evidence `docs/evidence/2026-10-15-br3-thin-etcd-witness.md`). Case (c) of `docs/replication-postgresql.md`, the old primary cut off (WireGuard dropped), then `keel database promote --old-primary-gone` on the standby: | | VIP on B | B out of recovery | First write acknowledged | From the VIP | | --- | --- | --- | --- | --- | | base | 1792062927.87 | 2939.45 | 2939.76 | 11.9 s | | netem 250 ms | 1792064189.10 | 4201.02 | 4201.99 | 12.9 s | | 2026-10-14, no etcd | | | | 0.83 s and 1.47 s | | CI | | | | 1.1 to 1.9 s | In both runs `keel-database-follow.path` on B was failed (keel#164), so the promotion came from the promote command's own follow step, after it told the peers (1.5 s bound for the old primary, then the witness). Before etcd, the path ran follow as soon as the VIP moved. The bound of 5 s from the VIP on B is not met. The cause is probably keel#164, but the follow step itself takes about 10 s here, and that is not explained yet. Re-measure after keel#164.