database: a MariaDB replica that rejoined or was reseeded stays VIP-fenced, so the pair fails over once only (keel#166 is PostgreSQL only) #192

New issue

Closed

opened 2026-10-16 07:26:44 -03:00 by navigator · 1 comment

navigator commented

2026-10-16 07:26:44 -03:00

Owner

Found in the rehearsal of the cross-site LNMP test, 2026-10-16 (keel 0.23.29, image lnmp 19.0-2). Pair: 9103 (BR2, first primary) and 9102 (BR1). Witness 9101 (BR1). etcd has 3 healthy voters.

A network cut of the primary 9103 moved the VIP to 9102 (epoch 2). After the heal, 9103 was diverged (see the acked loss issue), and keel spec apply --system-only --destroy-local-database reseeded it: "seeded from [fdf2:71:28d2::99c3]:3306 and replicating from it over TLS". The replica caught up (Seconds_Behind_Master: 0).
90 s later 9103 was still VIP-fenced:

role: replica
epoch 2, held by 3Pjf... at fdf2:71:28d2::99c3
fenced: this node lost it while cut off or away, and takes it again only by keel vip promote here

pct stop 9102 (the holder). 9101 and 9103 keep the etcd quorum. In 100 s, no node claimed the VIP. The writes stopped until the end of the test.
pct start 9102. 9102 also became fenced ("dropped, its lease is gone"; read only) and started to replicate from 9103 (Master_Host: fdf2:71:28d2::1bf7). Both nodes are fenced and read only, and each points at the other. No keel command except keel vip promote by the operator ends it.

Cause: the watch clears the VIP fence of a caught-up standby only for PostgreSQL (keel#166, keel/system/dbpgfollow.py _unfence). Nothing does it for a MariaDB replica that rejoined or was reseeded. So a MariaDB pair fails over automatically once only, as the PostgreSQL pair did before keel#166.

Expected: when the MariaDB replica replicates from the holder and has applied all that the holder has (GTID), the watch clears its fence, as for PostgreSQL.

Found in the rehearsal of the cross-site LNMP test, 2026-10-16 (keel 0.23.29, image lnmp 19.0-2). Pair: 9103 (BR2, first primary) and 9102 (BR1). Witness 9101 (BR1). etcd has 3 healthy voters. 1. A network cut of the primary 9103 moved the VIP to 9102 (epoch 2). After the heal, 9103 was diverged (see the acked loss issue), and `keel spec apply --system-only --destroy-local-database` reseeded it: "seeded from [fdf2:71:28d2::99c3]:3306 and replicating from it over TLS". The replica caught up (`Seconds_Behind_Master: 0`). 2. 90 s later 9103 was still VIP-fenced: ``` role: replica epoch 2, held by 3Pjf... at fdf2:71:28d2::99c3 fenced: this node lost it while cut off or away, and takes it again only by keel vip promote here ``` 3. `pct stop 9102` (the holder). 9101 and 9103 keep the etcd quorum. In 100 s, no node claimed the VIP. The writes stopped until the end of the test. 4. `pct start 9102`. 9102 also became fenced ("dropped, its lease is gone"; read only) and started to replicate from 9103 (`Master_Host: fdf2:71:28d2::1bf7`). Both nodes are fenced and read only, and each points at the other. No keel command except `keel vip promote` by the operator ends it. Cause: the watch clears the VIP fence of a caught-up standby only for PostgreSQL (keel#166, `keel/system/dbpgfollow.py` `_unfence`). Nothing does it for a MariaDB replica that rejoined or was reseeded. So a MariaDB pair fails over automatically once only, as the PostgreSQL pair did before keel#166. Expected: when the MariaDB replica replicates from the holder and has applied all that the holder has (GTID), the watch clears its fence, as for PostgreSQL.

navigator referenced this issue from a pull request that will close it,

2026-10-16 07:51:36 -03:00

fix: the VIP fence of a caught-up MariaDB replica is cleared (keel#192) #194

keel-bridge commented

2026-10-16 10:12:51 -03:00

Member

The bridge mirrored this item to GitHub: https://github.com/Keel-Linux/keel/issues/163
Comments, labels and the open or closed state sync both ways.

The bridge mirrored this item to GitHub: https://github.com/Keel-Linux/keel/issues/163 Comments, labels and the open or closed state sync both ways.

navigator closed this issue

2026-10-16 11:40:17 -03:00