fix: a MariaDB holder whose lease is in doubt acknowledges no write (keel#195) #197

Merged

navigator merged 4 commits from fix/mariadb-fence-on-lease-doubt-195 into main 2026-10-16 11:54:59 -03:00

Conversation 0 Commits 4 Files changed 13 +367 -11

navigator commented 2026-10-16 10:04:38 -03:00

Owner

Copy link

Fixes #195. Stacked on #194 → #191 → #190 → #189 → #187: merge those first.

Rehearsal 2 of the cross-site LNMP test (keel 0.23.38): on the cut MariaDB holder, a write that waited for the replica returned OK when the dump thread to the cut replica ended, 15.1 s after the cut (wait_no_slave = OFF, tcp_retries2 = 5), before the release (15 s from the last confirmed renewal). The VIP had left 1.7 s before, so by luck the OK did not reach the client; a client on the local socket gets it. The fence came from the watch, 35 s after the VIP left, because the release on a cut holder writes no state.

What changes:

• keel vip control marks, through its root half (bridge op doubt), the doubt of its own lease once the last renewal that the majority confirmed is 6 s old (DOUBT): /var/lib/keel/vip/<vip>.unconfirmed. It keeps the VIP until the release, as before, and removes the mark at the next confirmed renewal; a mark an earlier controller left goes at the first renewal the new one confirms; a controller that starts cut off doubts too; a mark goes with the claim. The mark is best effort and never stops the release. The root half accepts it for the node's own VIP only.
• keel database follow fences a MariaDB holder in doubt (read_only and the connections ended, #186/#193) and leaves the role; keel database watch follows a writable holder in doubt and leaves a fenced one. PostgreSQL is not changed: its commits wait.
• wait_no_slave = ON is not used: a new primary has no replica yet, and its first write would wait the whole timeout.
• docs: docs/vip.md (the holder's doubt), docs/replication.md (why the fence starts at the doubt; the 30 s timeout now justified by the measured fence time; and a correction: the firewall accepts 3306 on wg0 from every member, as for every mesh port).

Tests: controller (mark after DOUBT, cleared at the next renewal, stale mark cleared, restart cut off, mark goes with the claim, a mark that cannot be written never stops the release), bridge (mark, clear, refusal of what is not this node's VIP), follow and watch. The new tests fail without the fix (checked by the review).

Real nodes, rehearsal 3 (lnmp 19.0-2 with this keel installed by hand, BR1 + BR2): cut to read only 8.8 s (base) and 6.1 s (netem 250 ms); the dump thread ended at the fence, not before; 0 of 6908 and 0 of 13337 acknowledged ids lost with a local socket writer and a VIP writer; 0 spurious marks in 2 quiet windows of 10 min (base and netem). Raw notes stay with the evidence of the release test.

keel 0.23.39.

Fixes #195. Stacked on #194 → #191 → #190 → #189 → #187: merge those first. Rehearsal 2 of the cross-site LNMP test (keel 0.23.38): on the cut MariaDB holder, a write that waited for the replica returned OK when the dump thread to the cut replica ended, 15.1 s after the cut (`wait_no_slave = OFF`, `tcp_retries2 = 5`), before the release (15 s from the last confirmed renewal). The VIP had left 1.7 s before, so by luck the OK did not reach the client; a client on the local socket gets it. The fence came from the watch, 35 s after the VIP left, because the release on a cut holder writes no state. What changes: - `keel vip control` marks, through its root half (bridge op `doubt`), the doubt of its own lease once the last renewal that the majority confirmed is 6 s old (`DOUBT`): `/var/lib/keel/vip/<vip>.unconfirmed`. It keeps the VIP until the release, as before, and removes the mark at the next confirmed renewal; a mark an earlier controller left goes at the first renewal the new one confirms; a controller that starts cut off doubts too; a mark goes with the claim. The mark is best effort and never stops the release. The root half accepts it for the node's own VIP only. - `keel database follow` fences a MariaDB holder in doubt (read_only and the connections ended, #186/#193) and leaves the role; `keel database watch` follows a writable holder in doubt and leaves a fenced one. PostgreSQL is not changed: its commits wait. - `wait_no_slave = ON` is not used: a new primary has no replica yet, and its first write would wait the whole timeout. - docs: `docs/vip.md` (the holder's doubt), `docs/replication.md` (why the fence starts at the doubt; the 30 s timeout now justified by the measured fence time; and a correction: the firewall accepts 3306 on wg0 from every member, as for every mesh port). Tests: controller (mark after DOUBT, cleared at the next renewal, stale mark cleared, restart cut off, mark goes with the claim, a mark that cannot be written never stops the release), bridge (mark, clear, refusal of what is not this node's VIP), follow and watch. The new tests fail without the fix (checked by the review). Real nodes, rehearsal 3 (lnmp 19.0-2 with this keel installed by hand, BR1 + BR2): cut to read only 8.8 s (base) and 6.1 s (netem 250 ms); the dump thread ended at the fence, not before; 0 of 6908 and 0 of 13337 acknowledged ids lost with a local socket writer and a VIP writer; 0 spurious marks in 2 quiet windows of 10 min (base and netem). Raw notes stay with the evidence of the release test. keel 0.23.39.

navigator added 19 commits 2026-10-16 10:04:38 -03:00

test: the other member of a pair gets the primary's Anubis key (common#56) ... 59fbb4bdf5

On a pair, keel-overlay-anubis-key.service makes no key on the database
replica: the key is shared and comes from the primary (decision 0041).
Nothing copied it, so anubis@keel did not start there, and the apply
failed the anubis overlay. The LNMP and LNPP pair tests copied the key
by hand (Keel-Linux/common#56).

The tests ask that the other member of the pair record, and no other
member, gets the key over the members' channel; that a paired replica
with no key asks for it in a run that may change the machine, and
writes it with mode 0600 before Anubis starts; that a replica in no
pair yet starts nothing and says why; and that an unreachable primary
refuses the step.

fix: a paired replica asks the other member for Anubis's key (common#56) ... 257b2b2ea4

The secret message of the members' channel gives a second name,
anubis/signing_key, answered with /etc/anubis/keel.key only to the
other member of the pair record, as the replication credential is.

keel.system.anubiskey observes, on the live system and only in a run
that may change the machine, a node whose spec makes it a database
replica, with Anubis enabled, no key file and no
secrets.anubis_signing_key declared. It asks the other member of its
pair, and the plan writes the key (mode 0600) before the overlay's
units start. The key unit then finds the key and leaves it.

Before the pair record exists, the apply starts nothing for Anubis and
says why, so the first boot of a replica no longer fails the overlay. A
key that is there is never replaced.

chore: keel 0.23.34 (common#56) 5adbb9dc61

test: the VIP overlay waits for etcd's cluster with no drift (keel#106) ... eca601c3ce

keel-vip.service starts only when /var/lib/keel/etcd/cluster.json
exists. On a cloud simple pair, keel diff reported overlays.vip as
drift (enabled and inactive) on every run, and each apply printed a
start of the unit that did nothing.

The tests ask that, before the cluster exists, diff compares only
whether the unit is enabled and says why it does not run, and that
apply enables the unit and does not start it. With the cluster, an
inactive controller is drift and apply starts it, as before.

fix: the VIP overlay waits for etcd's cluster, as etcd does (keel#106) ... 6fa5d779a6

Before etcd's cluster.json exists (always in cloud simple, and in
cloud advanced until etcd forms), the start condition of
keel-vip.service holds it. keel diff now compares only whether the
unit is enabled, and gives the reason: without etcd, keel vip check
(keel-vip-check.timer) does the VIP's work. keel spec apply enables the
unit, does not start it, and says why. With the cluster, both compare
and converge the unit as before.

chore: keel 0.23.35 (keel#106) 7df4bc9649

test: the fence of an old MariaDB primary, on two real servers (keel#186, keel#193) ... cbadcf4cdd

On a cut MariaDB primary, a write that waits for the replica's ACK
returned OK when the semi-synchronous timeout (10 s) ended, before the
old primary dropped the VIP (15 s). The fence did not stop it either:
SET GLOBAL read_only = ON waits for the backup lock behind that write.
So writes that the old primary acknowledged were lost at a failover.

A dump thread to the cut replica also stayed on the old primary. When
that replica later connected again with the same server id, MariaDB
11.8.6 ended the old thread and took no connection after it.

tests/test_mariadb_fence_real.py runs two real mariadbd, semi-
synchronous with AFTER_SYNC. With the replica stopped by SIGSTOP, the
fence must end with read_only on, well within the timeout, with an
error and no OK for the waiting write. With the replica behind a relay
that stops moving bytes and keeps its sockets, the fence must end the
dump thread, and the server must answer after the replica connects
again with the same server id. The timeout must be at least twice the
VIP's release. The job "mariadb / trixie" runs it as root.

fix: the fence of an old MariaDB primary ends its connections (keel#186, keel#193) ... 14d982a579

keel database follow turns a node read only that took writes when it
loses the VIP. It now sets read_only in a client of its own and, while
that waits for the backup lock, kills every connection but the
server's own threads and this node's own replication threads. A write
that waited for the replica's ACK gets an error, not OK. Its row stays
on the old primary as an errant transaction, so the node comes back
diverged and waits for the confirmed reseed, as before. The dump
threads of its replicas end too: a fenced node serves no replica, and
a KILL ends such a thread cleanly where the implicit end at a new
connection of the same server id hung the server (keel#193).

rpl_semi_sync_master_timeout goes from 10 s to 30 s: longer than the
VIP's release (15 s, keel#174) and the follow it starts, with room. A
lost replica still gives the fallback to asynchronous commits, 30 s
after the last ACK in place of 10 s. This changes the 10 s of the
maintainer's answer of 2026-10-03 (decision 0031): the release test's
criterion is no acknowledged write lost at a cut of the primary, and
10 s cannot meet it while the release is 15 s.

chore: keel 0.23.36 (keel#186, keel#193) 98bcface9d

test: etcd that an apply starts is recorded, and the start enables wireguard (keel#188) ... a4386c6933

The cross-site rehearsal of 2026-10-16 booted 3 cloud advanced CTs with
overlays.wireguard and overlays.etcd disabled. At the third join, the
start of etcd wrote overlays.etcd: enabled into a spec that still said
wireguard: disabled, which does not validate, and etcd did not start.
The operator then enabled both and applied: etcd started, but nothing
wrote /var/lib/keel/etcd/started, and every later apply refused the
member as one keel never started.

The tests ask that the start enables wireguard with etcd, and that an
apply that starts etcd for the cluster of cluster.json records it.

fix: an apply that starts etcd records it, and the start enables wireguard (keel#188) ... 2109aef4f6

keel.mesh.etcd.start sets overlays.wireguard: enabled with
overlays.etcd: enabled. etcd requires wireguard, and the join already
wrote network.overlay.

plan_etcd writes /var/lib/keel/etcd/started on the live system when
the cluster is known and etcd holds no member yet, as the join and the
form do when they start etcd. A member directory that keel did not
start is still refused.

chore: keel 0.23.37 (keel#188) 1a5aa16f3a

test: a MariaDB replica that caught up with the holder is unfenced (keel#192) ... e1eeadee44

The cross-site rehearsal of 2026-10-16 cut the MariaDB primary, and
the VIP moved by itself. The old primary was reseeded and replicated
from the holder, caught up, but stayed VIP-fenced: the clear of
keel#166 is PostgreSQL only. A pct stop of the holder then gave no
failover, and after pct start both nodes were fenced and read only.

The tests ask that follow clears the fence of a replica that replicates
from the holder, with its SQL thread running, once it applied the
newest sequence the holder logged in each GTID domain, also when a
reseed set it in the slave position only; that a replica behind, one
whose SQL thread stopped, or one whose holder does not answer stays
fenced; and that the watch follows a fenced MariaDB replica.

fix: the VIP fence of a caught-up MariaDB replica is cleared (keel#192) ... 54d094b4c3

keel database follow clears the VIP fence of a MariaDB replica that
replicates from the holder, with its SQL thread running, once it
applied the newest sequence that the holder logged in each GTID
domain: in its binary log (log_slave_updates) or its slave position,
which a reseed sets from the dump. This is keel#166 for MariaDB. The
watch runs follow on a fenced replica of either engine.

chore: keel 0.23.38 (keel#192) fb23385529

test: a MariaDB holder whose lease is in doubt is fenced (keel#195) ... 2a63b0c7ef

Rehearsal 2 of the cross-site LNMP test, 2026-10-16, with the fix of
keel#186: on the cut MariaDB holder, a write that waited for the
replica returned OK when the dump thread to the cut replica ended,
15.1 s after the cut (wait_no_slave = OFF, tcp_retries2 = 5). The VIP
had left 1.7 s before, so the OK did not reach the client by luck. The
fence came from the watch, 35 s after the VIP left: the release on a
cut holder writes no state.

The tests ask that the controller marks the doubt of its lease once
its last confirmed renewal is DOUBT s old, keeps the VIP, and clears
the mark at the next confirmed renewal, also a mark an earlier
controller left; that the root half refuses a mark for what is not a
VIP; that follow fences a MariaDB holder while the mark is there and
lifts read_only once it is gone; that the watch follows a writable
holder in doubt and leaves a fenced one; and that DOUBT plus a follow
ends well before a dump thread to a cut replica can end.

fix: a MariaDB holder whose lease is in doubt acknowledges no write (keel#195) ... 50578dc4bf

keel vip control marks, through its root half, the doubt of the lease
of its own claim once its last renewal that the majority confirmed is
6 s old (DOUBT), in /var/lib/keel/vip/<vip>.unconfirmed, and removes
the mark at the next confirmed renewal. The holder keeps the VIP until
the release, as before. keel-database-follow.path starts follow, which
fences a MariaDB holder in doubt (read_only and the client connections
ended, keel#186) and leaves the role; the watch does not lift it.

On a cut holder, a write that waits for the replica returned OK when
the dump thread to the cut replica ended, about 12.6 s after the cut,
before the release at 15 s. 6 s is above the longest normal gap
between two confirmed renewals (4 s). wait_no_slave = ON is not used:
a new primary has no replica yet, and its first write would wait for
the whole timeout.

docs: the MariaDB fence at the doubt, and what the firewall accepts on wg0 (keel#195) ... 5f3d649e79

docs/replication.md says why the fence of a MariaDB holder starts at
the doubt of its lease. It also corrects one sentence: the derived
firewall accepts 3306 on wg0 from every member of the mesh, as for
every mesh port (0041), not from the pair peer and for the VIP only.
The real tests of 2026-10-16 found the difference.

chore: keel 0.23.39 (keel#195) 9c0b9e8a59

navigator referenced this pull request 2026-10-16 10:04:49 -03:00

fix: the fence of an old MariaDB primary ends its connections (keel#186, keel#193) #190

navigator referenced this pull request from Keel-Linux/handbook 2026-10-16 10:05:33 -03:00

docs: amend 0031 and 0049, the MariaDB timeout and the fence at the doubt of the lease #98

navigator merged commit d18ed23fac into main 2026-10-16 11:54:59 -03:00

navigator referenced this pull request from a commit 2026-10-16 11:54:59 -03:00

Merge pull request 'fix: a MariaDB holder whose lease is in doubt acknowledges no write (keel#195)' (#197) from fix/mariadb-fence-on-lease-doubt-195 into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

accessibility

Barrier affecting people with disabilities

  

bug

Something isn't working

  

documentation

Improvements or additions to documentation

  

duplicate

This issue or pull request already exists

  

enhancement

New feature or request

  

good first issue

Good for newcomers

  

help wanted

Extra attention is needed

  

invalid

This doesn't seem right

  

question

Further information is requested

  

wontfix

This will not be worked on

No labels

accessibility

bug

documentation

duplicate

enhancement

good first issue

help wanted

invalid

question

wontfix

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

Keel-Linux/keel!197

No description provided.