database: mariadbd on a reseeded MariaDB replica accepts no connection after the old holder connects again with the same server id #193

New issue

Closed

opened 2026-10-16 07:26:44 -03:00 by navigator · 1 comment

navigator commented

2026-10-16 07:26:44 -03:00

Owner

Found in the rehearsal of the cross-site LNMP test, 2026-10-16 (keel 0.23.29, MariaDB 11.8.6-0+deb13u1, image lnmp 19.0-2). Same run as the unfence issue.

Sequence on 9103 (BR2):

10:12:34 the network of 9103 (then the primary) was cut (nft drops UDP 51820). Its semi-sync binlog dump thread 278 to the replica 9102 (server id 1885343376) stayed. It did not end during the 91 s cut, nor after the heal.
10:16:32 9103 was reseeded and replicated from 9102.
10:18:38 pct stop 9102. 10:20:41 9102 started again. As a fenced node, 9102 replicated from 9103 with the same server id.
10:21:37 9103 logged "Stop semi-sync binlog_dump to slave (server_id: 1885343376)" for the old thread 278.
After that, mariadbd on 9103 accepts no new connection, by the socket or by TCP (mariadb -e 'select 1' times out after 5 s). keel database watch hangs on SHOW REPLICA STATUS. 30 one_connection threads wait on a futex.

Threads: 42
  30 one_connection futex_do_wait
   1 slave_io poll_schedule_timeout
   1 Ack_receiver poll_schedule_timeout

Possible causes to look at:

A dump thread to a cut replica lives for minutes. slave_net_timeout is 10 s, but the master side has no bound on a blocked write (net_write_timeout, TCP retransmission).
The kill of the old dump thread, when a replica with the same server id connects, deadlocks with semi-sync.

keel should keep a dump thread to a cut replica from living this long, or keel database follow should not make the old holder replicate from a fenced node (see the unfence issue).

Found in the rehearsal of the cross-site LNMP test, 2026-10-16 (keel 0.23.29, MariaDB 11.8.6-0+deb13u1, image lnmp 19.0-2). Same run as the unfence issue. Sequence on 9103 (BR2): - 10:12:34 the network of 9103 (then the primary) was cut (nft drops UDP 51820). Its semi-sync binlog dump thread 278 to the replica 9102 (server id 1885343376) stayed. It did not end during the 91 s cut, nor after the heal. - 10:16:32 9103 was reseeded and replicated from 9102. - 10:18:38 `pct stop 9102`. 10:20:41 9102 started again. As a fenced node, 9102 replicated from 9103 with the same server id. - 10:21:37 9103 logged "Stop semi-sync binlog_dump to slave (server_id: 1885343376)" for the old thread 278. - After that, mariadbd on 9103 accepts no new connection, by the socket or by TCP (`mariadb -e 'select 1'` times out after 5 s). `keel database watch` hangs on `SHOW REPLICA STATUS`. 30 `one_connection` threads wait on a futex. ``` Threads: 42 30 one_connection futex_do_wait 1 slave_io poll_schedule_timeout 1 Ack_receiver poll_schedule_timeout ``` Possible causes to look at: - A dump thread to a cut replica lives for minutes. `slave_net_timeout` is 10 s, but the master side has no bound on a blocked write (net_write_timeout, TCP retransmission). - The kill of the old dump thread, when a replica with the same server id connects, deadlocks with semi-sync. keel should keep a dump thread to a cut replica from living this long, or keel database follow should not make the old holder replicate from a fenced node (see the unfence issue).

navigator referenced this issue from a pull request that will close it,

2026-10-16 07:48:21 -03:00

fix: the fence of an old MariaDB primary ends its connections (keel#186, keel#193) #190

keel-bridge commented

2026-10-16 10:12:48 -03:00

Member

The bridge mirrored this item to GitHub: https://github.com/Keel-Linux/keel/issues/162
Comments, labels and the open or closed state sync both ways.

The bridge mirrored this item to GitHub: https://github.com/Keel-Linux/keel/issues/162 Comments, labels and the open or closed state sync both ways.

navigator closed this issue

2026-10-16 11:09:54 -03:00