database: a cut MariaDB holder acknowledges a write when the dump thread ends, before the fence (no VIP state at the release, wait_no_slave OFF) #195

New issue

Closed

opened 2026-10-16 08:23:14 -03:00 by navigator · 1 comment

navigator commented

2026-10-16 08:23:14 -03:00

Owner

A network cut of the primary 9103 (nft drops UDP 51820). 0 of 890 acknowledged ids were lost, but only by luck of 1.7 s. Timeline on 9103:

9466.74  cut
9466.80  write 200084 starts on 9103 through the VIP, waits for the replica's ACK
9480.16  the VIP leaves wg0 of 9103 (its lifetime ends)
9481.85  mariadbd: Stop semi-sync binlog_dump to slave ... Aborted connection ... user 'repl' (Got an error writing communication packets)
9494.94  mariadbd: Aborted connection 205 ... user 'pt' host 'fd6d:829f:348b::ffff:1' (Got timeout reading communication packets)
9515.46  keel database watch: "set read_only ON ... every client connection was ended (keel#186)"
9559.17  the heal; the first write of /var/lib/keel/vip/<vip>.json since the cut

Two problems:

The release on a cut holder writes no VIP state. The kernel ends the address at its lifetime, and nothing writes /var/lib/keel/vip/. So keel-database-follow.path (PathChanged=/var/lib/keel/vip) does not start follow. The fence of keel#186 came from keel-database-watch.timer, 35.3 s after the VIP left. That is later than the semi-synchronous timeout (30 s) from the start of the write.
rpl_semi_sync_master_wait_no_slave = OFF releases the waiting write when the dump thread ends. The dump thread to the cut replica ended 15.1 s after the cut (net.ipv4.tcp_retries2 = 5). With no replica left, MariaDB returned OK for write 200084 at about 9481.85. The row is on 9103 only (an errant GTID). The client did not get the OK only because the VIP had left wg0 1.7 s before and the answer could not reach it. A client on the local socket (PHP on the same node) gets that OK: an acknowledged write that the new primary does not have.

So the fix of keel#186 is not enough: the fence must come before the dump thread to a cut replica ends, or a commit must not return OK when the last replica goes. Possible fixes, to decide:

the holder that loses its renewal turns the database read only and ends the client connections itself (the controller writes the state when it drops the VIP, so the path starts follow at once);
rpl_semi_sync_master_wait_no_slave = ON, so the waiting commit waits for the timeout and not for the dump thread (the replica's own SQL thread then needs rpl_semi_sync_master_enabled = OFF while it is a replica, see docs/replication.md).

Found in rehearsal 2 of the cross-site LNMP test, 2026-10-16: image lnmp 19.0-2 with keel 0.23.38 from the open fix stack (keel#187, #189, #190, #191, #194). Pair 9103 (BR2, primary) and 9102 (BR1), witness 9101 (BR1), etcd 3 voters. Times are Unix seconds (`9466.74` means `1792149466.74`). A network cut of the primary 9103 (nft drops UDP 51820). 0 of 890 acknowledged ids were lost, but only by luck of 1.7 s. Timeline on 9103: ``` 9466.74 cut 9466.80 write 200084 starts on 9103 through the VIP, waits for the replica's ACK 9480.16 the VIP leaves wg0 of 9103 (its lifetime ends) 9481.85 mariadbd: Stop semi-sync binlog_dump to slave ... Aborted connection ... user 'repl' (Got an error writing communication packets) 9494.94 mariadbd: Aborted connection 205 ... user 'pt' host 'fd6d:829f:348b::ffff:1' (Got timeout reading communication packets) 9515.46 keel database watch: "set read_only ON ... every client connection was ended (keel#186)" 9559.17 the heal; the first write of /var/lib/keel/vip/<vip>.json since the cut ``` Two problems: 1. **The release on a cut holder writes no VIP state.** The kernel ends the address at its lifetime, and nothing writes `/var/lib/keel/vip/`. So `keel-database-follow.path` (PathChanged=/var/lib/keel/vip) does not start follow. The fence of keel#186 came from `keel-database-watch.timer`, 35.3 s after the VIP left. That is later than the semi-synchronous timeout (30 s) from the start of the write. 2. **`rpl_semi_sync_master_wait_no_slave = OFF` releases the waiting write when the dump thread ends.** The dump thread to the cut replica ended 15.1 s after the cut (`net.ipv4.tcp_retries2 = 5`). With no replica left, MariaDB returned OK for write 200084 at about 9481.85. The row is on 9103 only (an errant GTID). The client did not get the OK only because the VIP had left wg0 1.7 s before and the answer could not reach it. A client on the local socket (PHP on the same node) gets that OK: an acknowledged write that the new primary does not have. So the fix of keel#186 is not enough: the fence must come before the dump thread to a cut replica ends, or a commit must not return OK when the last replica goes. Possible fixes, to decide: - the holder that loses its renewal turns the database read only and ends the client connections itself (the controller writes the state when it drops the VIP, so the path starts follow at once); - `rpl_semi_sync_master_wait_no_slave = ON`, so the waiting commit waits for the timeout and not for the dump thread (the replica's own SQL thread then needs `rpl_semi_sync_master_enabled = OFF` while it is a replica, see docs/replication.md).

navigator referenced this issue from a pull request that will close it,

2026-10-16 10:04:38 -03:00

fix: a MariaDB holder whose lease is in doubt acknowledges no write (keel#195) #197

keel-bridge commented

2026-10-16 10:12:45 -03:00

Member

The bridge mirrored this item to GitHub: https://github.com/Keel-Linux/keel/issues/161
Comments, labels and the open or closed state sync both ways.

The bridge mirrored this item to GitHub: https://github.com/Keel-Linux/keel/issues/161 Comments, labels and the open or closed state sync both ways.

navigator closed this issue

2026-10-16 11:54:59 -03:00