fix: the fence of an old MariaDB primary ends its connections (keel#186, keel#193) #190

Merged

navigator merged 3 commits from fix/mariadb-fence-kills-waiting-writes-186 into main 2026-10-16 11:09:54 -03:00

Conversation 0 Commits 3 Files changed 10 +515 -19

navigator commented 2026-10-16 07:12:49 -03:00

Owner

Copy link

Fixes #186. Fixes #193. Stacked on #189 (keel#106) → #187 → main: merge those first. The fence of this PR alone comes at the release; #197 moves it to the doubt of the lease.

On a cut MariaDB primary, acknowledged writes were lost at a failover (measured in the cross-site rehearsal: 12 of 1906 acknowledged ids lost, see #186):

1. rpl_semi_sync_master_timeout = 10000 is shorter than the VIP's release (15 s, keel#174). A write that waits for the replica's ACK returned OK after 10 s, and the next writes returned OK at once, until the old primary dropped the VIP.
2. The fence did not stop them: SET GLOBAL read_only = ON waits for the backup lock behind the waiting write, which then returns OK when the timeout ends.

And #193: a dump thread to the cut replica stayed on the old primary. When that replica later connected again with the same server id, MariaDB 11.8.6 ended the old thread and then took no connection at all. Reproduced locally with a relay that stops moving bytes and keeps its sockets; a KILL of the stale thread ends it cleanly, and the reconnect then works.

What changes:

• keel.system.dbreadonly.fence: SET GLOBAL read_only = ON in a client of its own (waits until its statement shows, sent again if a kill ended it), and while it waits, KILL CONNECTION of every connection but the server's own threads and this node's own replication threads; one more pass after it. A waiting write gets an error and no OK; the dump threads of its replicas end, because a fenced node serves no replica. keel database follow uses it when it turns a node read only that took writes.
• rpl_semi_sync_master_timeout = 30000. With #197 (keel#195) the fence starts at the doubt of the lease; rehearsal 3 measured the cut to read only at 6.1 to 8.8 s, so 10 s would leave 1.2 s in the worst case and 30 s leaves room. A lost replica still gives the fallback, 30 s after the last ACK. This changes the 10 s of the maintainer's answer of 2026-10-03 (0031); the handbook amendment records it.
• docs/replication.md says it.

Tests: tests/test_mariadb_fence_real.py on two real mariadbd (semi-synchronous, AFTER_SYNC): the waiting write gets no OK, read_only is on well within the timeout, a new write is refused; with the black-hole relay, the dump thread ends and the server answers after the replica connects again with the same server id. Runs in "mariadb / trixie" as root (also as follow runs it, no seams). Without the kills the tests fail (checked).

keel 0.23.36.

Fixes #186. Fixes #193. Stacked on #189 (keel#106) → #187 → main: merge those first. The fence of this PR alone comes at the release; #197 moves it to the doubt of the lease. On a cut MariaDB primary, acknowledged writes were lost at a failover (measured in the cross-site rehearsal: 12 of 1906 acknowledged ids lost, see #186): 1. `rpl_semi_sync_master_timeout = 10000` is shorter than the VIP's release (15 s, keel#174). A write that waits for the replica's ACK returned OK after 10 s, and the next writes returned OK at once, until the old primary dropped the VIP. 2. The fence did not stop them: `SET GLOBAL read_only = ON` waits for the backup lock behind the waiting write, which then returns OK when the timeout ends. And #193: a dump thread to the cut replica stayed on the old primary. When that replica later connected again with the same server id, MariaDB 11.8.6 ended the old thread and then took no connection at all. Reproduced locally with a relay that stops moving bytes and keeps its sockets; a `KILL` of the stale thread ends it cleanly, and the reconnect then works. What changes: - `keel.system.dbreadonly.fence`: `SET GLOBAL read_only = ON` in a client of its own (waits until its statement shows, sent again if a kill ended it), and while it waits, `KILL CONNECTION` of every connection but the server's own threads and this node's own replication threads; one more pass after it. A waiting write gets an error and no OK; the dump threads of its replicas end, because a fenced node serves no replica. `keel database follow` uses it when it turns a node read only that took writes. - `rpl_semi_sync_master_timeout = 30000`. With #197 (keel#195) the fence starts at the doubt of the lease; rehearsal 3 measured the cut to read only at 6.1 to 8.8 s, so 10 s would leave 1.2 s in the worst case and 30 s leaves room. A lost replica still gives the fallback, 30 s after the last ACK. This changes the 10 s of the maintainer's answer of 2026-10-03 (0031); the handbook amendment records it. - `docs/replication.md` says it. Tests: `tests/test_mariadb_fence_real.py` on two real `mariadbd` (semi-synchronous, AFTER_SYNC): the waiting write gets no OK, `read_only` is on well within the timeout, a new write is refused; with the black-hole relay, the dump thread ends and the server answers after the replica connects again with the same server id. Runs in "mariadb / trixie" as root (also as follow runs it, no seams). Without the kills the tests fail (checked). keel 0.23.36.

navigator added 12 commits 2026-10-16 07:12:49 -03:00

test: a reseed works when repl has no CONNECT on a database (keel#183) ... ecad9eaa5a

The reseed probe sums pg_database_size() over every database as the
role repl. That function needs CONNECT on the database. The keel-lnpp
check database revoked CONNECT from PUBLIC, and every reseed of the
LNPP pair refused: "permission denied for database keel_check".

The netns test of the pair now revokes CONNECT on the shop database
from PUBLIC before case (d), as the check database did. The unit tests
ask that the probe measures only the databases repl may connect to,
and that a refusal for lack of space names the databases it left out.

fix: the reseed probe measures only the databases repl may connect to (keel#183) ... e33d7f3a5b

pg_database_size() needs CONNECT on the database or the role
pg_read_all_stats. An application database with REVOKE CONNECT FROM
PUBLIC stopped every reseed with "permission denied".

The probe now sums the size with a FILTER on
has_database_privilege(oid, 'CONNECT') and counts the databases it
leaves out. The free space check uses the size it has. When the check
refuses, the message says how many databases the need leaves out. The
base backup itself is not changed: it copies the whole cluster.

pg_read_all_stats is not granted to repl: it gives more read access
than the reseed needs.

chore: keel 0.23.33 (keel#183) a6271bbc9b

test: the other member of a pair gets the primary's Anubis key (common#56) ... fa7c2f14a9

On a pair, keel-overlay-anubis-key.service makes no key on the database
replica: the key is shared and comes from the primary (decision 0041).
Nothing copied it, so anubis@keel did not start there, and the apply
failed the anubis overlay. The LNMP and LNPP pair tests copied the key
by hand (Keel-Linux/common#56).

The tests ask that the other member of the pair record, and no other
member, gets the key over the members' channel; that a paired replica
with no key asks for it in a run that may change the machine, and
writes it with mode 0600 before Anubis starts; that a replica in no
pair yet starts nothing and says why; and that an unreachable primary
refuses the step.

fix: a paired replica asks the other member for Anubis's key (common#56) ... c8ccb7fa98

The secret message of the members' channel gives a second name,
anubis/signing_key, answered with /etc/anubis/keel.key only to the
other member of the pair record, as the replication credential is.

keel.system.anubiskey observes, on the live system and only in a run
that may change the machine, a node whose spec makes it a database
replica, with Anubis enabled, no key file and no
secrets.anubis_signing_key declared. It asks the other member of its
pair, and the plan writes the key (mode 0600) before the overlay's
units start. The key unit then finds the key and leaves it.

Before the pair record exists, the apply starts nothing for Anubis and
says why, so the first boot of a replica no longer fails the overlay. A
key that is there is never replaced.

chore: keel 0.23.34 (common#56) 240512d44c

test: the VIP overlay waits for etcd's cluster with no drift (keel#106) ... 6b44929057

keel-vip.service starts only when /var/lib/keel/etcd/cluster.json
exists. On a cloud simple pair, keel diff reported overlays.vip as
drift (enabled and inactive) on every run, and each apply printed a
start of the unit that did nothing.

The tests ask that, before the cluster exists, diff compares only
whether the unit is enabled and says why it does not run, and that
apply enables the unit and does not start it. With the cluster, an
inactive controller is drift and apply starts it, as before.

fix: the VIP overlay waits for etcd's cluster, as etcd does (keel#106) ... f53f31fe45

Before etcd's cluster.json exists (always in cloud simple, and in
cloud advanced until etcd forms), the start condition of
keel-vip.service holds it. keel diff now compares only whether the
unit is enabled, and gives the reason: without etcd, keel vip check
(keel-vip-check.timer) does the VIP's work. keel spec apply enables the
unit, does not start it, and says why. With the cluster, both compare
and converge the unit as before.

chore: keel 0.23.35 (keel#106) b1f0252597

test: a write that waits for the ACK gets no OK once the fence ran (keel#186) ... 60027d2d7f

On a cut MariaDB primary, a write that waits for the replica's ACK
returned OK when the semi-synchronous timeout (10 s) ended, before the
old primary dropped the VIP (15 s). The fence did not stop it either:
SET GLOBAL read_only = ON waits for the backup lock behind that write.
So writes that the old primary acknowledged were lost at a failover.

tests/test_mariadb_fence_real.py runs two real mariadbd, semi-
synchronous with AFTER_SYNC, the replica stopped by SIGSTOP. The fence
must end with read_only on, well within the timeout, with an error and
no OK for the waiting write, and with the replica's dump thread kept.
The timeout must be at least twice the VIP's release. The job
"mariadb / trixie" runs it as root.

fix: the fence of an old MariaDB primary ends its client connections (keel#186) ... c8378d2f07

keel database follow turns a node read only that took writes when it
loses the VIP. It now sets read_only in a client of its own and, while
that waits for the backup lock, kills every client connection (not the
server's own threads, not the replication threads). A write that waited
for the replica's ACK gets an error, not OK. Its row stays on the old
primary as an errant transaction, so the node comes back diverged and
waits for the confirmed reseed, as before.

rpl_semi_sync_master_timeout goes from 10 s to 30 s: longer than the
VIP's release (15 s, keel#174) and the follow it starts, with room. A
lost replica still gives the fallback to asynchronous commits, 30 s
after the last ACK in place of 10 s. This changes the 10 s of the
maintainer's answer of 2026-10-03 (decision 0031): the release test's
criterion is no acknowledged write lost at a cut of the primary, and
10 s cannot meet it while the release is 15 s.

chore: keel 0.23.36 (keel#186) 4a6fa9907d

navigator force-pushed fix/mariadb-fence-kills-waiting-writes-186 from 4a6fa9907d to c2fc854a51 2026-10-16 07:18:42 -03:00 Compare

navigator referenced this pull request 2026-10-16 07:24:18 -03:00

fix: an apply that starts etcd records it, and the start enables wireguard (keel#188) #191

navigator force-pushed fix/mariadb-fence-kills-waiting-writes-186 from c2fc854a51 to 81858ed17a 2026-10-16 07:47:28 -03:00 Compare

navigator force-pushed fix/mariadb-fence-kills-waiting-writes-186 from 81858ed17a to e3c6afa214 2026-10-16 07:48:07 -03:00 Compare

navigator changed title from fix: the fence of an old MariaDB primary ends its client connections (keel#186) to fix: the fence of an old MariaDB primary ends its connections (keel#186, keel#193) 2026-10-16 07:48:21 -03:00

navigator referenced this pull request 2026-10-16 07:51:36 -03:00

fix: the VIP fence of a caught-up MariaDB replica is cleared (keel#192) #194

navigator force-pushed fix/mariadb-fence-kills-waiting-writes-186 from e3c6afa214 to ebe37aee59 2026-10-16 07:55:00 -03:00 Compare

navigator referenced this pull request 2026-10-16 08:23:14 -03:00

database: a cut MariaDB holder acknowledges a write when the dump thread ends, before the fence (no VIP state at the release, wait_no_slave OFF) #195

navigator force-pushed fix/mariadb-fence-kills-waiting-writes-186 from ebe37aee59 to 98bcface9d 2026-10-16 10:04:20 -03:00 Compare

navigator referenced this pull request 2026-10-16 10:04:38 -03:00

fix: a MariaDB holder whose lease is in doubt acknowledges no write (keel#195) #197

navigator referenced this pull request from Keel-Linux/handbook 2026-10-16 10:05:33 -03:00

docs: amend 0031 and 0049, the MariaDB timeout and the fence at the doubt of the lease #98

navigator merged commit 4871fffaf4 into main 2026-10-16 11:09:54 -03:00

navigator referenced this pull request from a commit 2026-10-16 11:09:55 -03:00

Merge pull request 'fix: the fence of an old MariaDB primary ends its connections (keel#186, keel#193)' (#190) from fix/mariadb-fence-kills-waiting-writes-186 into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

accessibility

Barrier affecting people with disabilities

  

bug

Something isn't working

  

documentation

Improvements or additions to documentation

  

duplicate

This issue or pull request already exists

  

enhancement

New feature or request

  

good first issue

Good for newcomers

  

help wanted

Extra attention is needed

  

invalid

This doesn't seem right

  

question

Further information is requested

  

wontfix

This will not be worked on

No labels

accessibility

bug

documentation

duplicate

enhancement

good first issue

help wanted

invalid

question

wontfix

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

Keel-Linux/keel!190

No description provided.