database: a cut MariaDB primary acknowledges writes the new primary never gets (semi-sync timeout 10 s, fence waits behind the write) #186

New issue

Closed

opened 2026-10-16 07:03:03 -03:00 by navigator · 2 comments

navigator commented

2026-10-16 07:03:03 -03:00

Owner

Problem. On a MariaDB pair, a cut of the primary's site can lose writes that the old primary acknowledged. Two causes:

rpl_semi_sync_master_timeout = 10000 is shorter than the VIP release (RELEASE_AFTER = 15 s, keel#174, the 0049 amendment of 2026-10-15). A write that waits for the replica's ACK on the cut primary commits and returns OK after 10 s. Then semi-sync is OFF and the next writes return OK at once, until the old primary drops the VIP at about 15 s and turns read only. These writes are errant: the new primary does not have them.
Even with a longer timeout, the fence does not stop them. keel database follow sets read_only = ON on the node that lost the VIP. SET GLOBAL read_only = ON waits for the backup lock behind a write that waits for the ACK, and the write returns OK when the timeout ends:

id  state                                   info
19  Waiting for backup lock                 set global read_only=ON
18  Waiting for semi-sync ACK from slave    insert into t.a values (3)
client3 rc=0 at 10:02:32.057   (the write: OK, acknowledged)
ro rc=0 at 10:02:32.058        (read_only on only after it)

A KILL CONNECTION of the waiting write gives the client an error, not OK:

12  Waiting for semi-sync ACK from slave  insert into t.a values (2)
ERROR 2026 (HY000): TLS/SSL error: unexpected eof while reading   (client rc=1)

The row stays on the old primary (an errant GTID, so the old primary comes back diverged and needs the confirmed reseed), but no client was told it is stored.

What is needed (the least power):

The fence of the old primary kills the client connections before and while it sets read_only = ON, so no waiting write gets an OK after the node lost the VIP. Replication threads and keel's own session stay.
rpl_semi_sync_master_timeout longer than the VIP release plus the time follow needs, so the fence comes first. This changes the 10 s of the maintainer's answer of 2026-10-03 (0031). It keeps the fallback for a lost replica, only later.

The pass criterion of the release test is "0 acknowledged writes lost" for a network cut of the primary. Without this fix, the LNMP pair fails it by design.

Found while the cross-site release test of the LNMP pair was prepared, 2026-10-16. Reproduced on a local MariaDB 11.8.6 pair (semi-synchronous, AFTER_SYNC), with the replica stopped by SIGSTOP as a cut. **Problem.** On a MariaDB pair, a cut of the primary's site can lose writes that the old primary acknowledged. Two causes: 1. `rpl_semi_sync_master_timeout = 10000` is shorter than the VIP release (`RELEASE_AFTER = 15 s`, keel#174, the 0049 amendment of 2026-10-15). A write that waits for the replica's ACK on the cut primary commits and returns OK after 10 s. Then semi-sync is OFF and the next writes return OK at once, until the old primary drops the VIP at about 15 s and turns read only. These writes are errant: the new primary does not have them. 2. Even with a longer timeout, the fence does not stop them. `keel database follow` sets `read_only = ON` on the node that lost the VIP. `SET GLOBAL read_only = ON` waits for the backup lock behind a write that waits for the ACK, and the write returns OK when the timeout ends: ``` id state info 19 Waiting for backup lock set global read_only=ON 18 Waiting for semi-sync ACK from slave insert into t.a values (3) client3 rc=0 at 10:02:32.057 (the write: OK, acknowledged) ro rc=0 at 10:02:32.058 (read_only on only after it) ``` A `KILL CONNECTION` of the waiting write gives the client an error, not OK: ``` 12 Waiting for semi-sync ACK from slave insert into t.a values (2) ERROR 2026 (HY000): TLS/SSL error: unexpected eof while reading (client rc=1) ``` The row stays on the old primary (an errant GTID, so the old primary comes back diverged and needs the confirmed reseed), but no client was told it is stored. **What is needed** (the least power): 1. The fence of the old primary kills the client connections before and while it sets `read_only = ON`, so no waiting write gets an OK after the node lost the VIP. Replication threads and keel's own session stay. 2. `rpl_semi_sync_master_timeout` longer than the VIP release plus the time follow needs, so the fence comes first. This changes the 10 s of the maintainer's answer of 2026-10-03 (0031). It keeps the fallback for a lost replica, only later. The pass criterion of the release test is "0 acknowledged writes lost" for a network cut of the primary. Without this fix, the LNMP pair fails it by design.

navigator referenced this issue from a pull request that will close it,

2026-10-16 07:12:49 -03:00

fix: the fence of an old MariaDB primary ends its connections (keel#186, keel#193) #190

navigator commented

2026-10-16 07:27:01 -03:00

Author

Owner

Event	From the cut
id 200131 on 9103 acknowledged after 9.94 s ("Timeout waiting for reply of binlog ... Semi-sync replication switched OFF")	+9.99 s
ids 200132 to 200142 acknowledged on 9103, asynchronous	+10.19 s to +12.16 s
VIP off on 9103	+12.20 s
VIP on 9102 (epoch 2)	+20.16 s
first acknowledged write on 9102 (9 writes before it got 1290, read only)	+21.67 s, 1.50 s after the VIP
first acknowledged write of the 9101 client (its write to the cut 9103 hung 27 s, then TLS error)	+27.12 s

Acknowledged ids: 1906. On 9102: 1894. Missing: 200131 to 200142 (12), all acknowledged by the cut 9103. After the heal 9103 was diverged: errant 0-470210458-277 (and 0-1455937763-6, the server id of its first boot). The 9101 client lost nothing: its write in flight at the cut was not acknowledged.

Measured in the cross-site rehearsal, 2026-10-16 (keel 0.23.29, lnmp 19.0-2; primary 9103 on BR2, replica 9102 and witness 9101 on BR1, 5 ms RTT). Cut of 9103 at 5554.20 (nft drops UDP 51820). A writer on each pair node wrote through the VIP only while the node carried it; a third writer on 9101 wrote through the VIP all the time. | Event | From the cut | | --- | --- | | id 200131 on 9103 acknowledged after 9.94 s ("Timeout waiting for reply of binlog ... Semi-sync replication switched OFF") | +9.99 s | | ids 200132 to 200142 acknowledged on 9103, asynchronous | +10.19 s to +12.16 s | | VIP off on 9103 | +12.20 s | | VIP on 9102 (epoch 2) | +20.16 s | | first acknowledged write on 9102 (9 writes before it got 1290, read only) | +21.67 s, 1.50 s after the VIP | | first acknowledged write of the 9101 client (its write to the cut 9103 hung 27 s, then TLS error) | +27.12 s | Acknowledged ids: 1906. On 9102: 1894. Missing: 200131 to 200142 (12), all acknowledged by the cut 9103. After the heal 9103 was diverged: errant `0-470210458-277` (and `0-1455937763-6`, the server id of its first boot). The 9101 client lost nothing: its write in flight at the cut was not acknowledged.

keel-bridge commented

2026-10-16 10:12:57 -03:00

Member

The bridge mirrored this item to GitHub: https://github.com/Keel-Linux/keel/issues/165
Comments, labels and the open or closed state sync both ways.

The bridge mirrored this item to GitHub: https://github.com/Keel-Linux/keel/issues/165 Comments, labels and the open or closed state sync both ways.

navigator closed this issue

2026-10-16 11:09:54 -03:00