feat: MariaDB primary and replica on the mesh, semi-synchronous, the primary the VIP's holder (0031, 0049) #93

2026-10-08T11:09:17-03:00

marcos-mendez commented

2026-10-08 11:09:17 -03:00

(Migrated from github.com)

Phase 2, step 3: MariaDB primary and replica over the mesh, semi-synchronous, the primary the VIP's holder (decisions 0013, 0020, 0028, 0031, 0049, 0050; tracker#57; the maintainer's answers of 2026-10-03 and 2026-10-11). The design is docs/replication.md; the four questions it asked were approved as recommended on 2026-10-11 (the precision to 0049 is Keel-Linux/handbook#57). keel 0.23.0.

What it does

Pair model. appliance.vip plus database.server.role: the primary is the VIP's holder; role in the spec is the installation's, decides who seeds from whom before the first claim, and is runtime state from then on (0020). inspect reports the VIP's role; diff shows the difference as information, never drift; apply never converges it. With the pair record, replication.primary, allowed_from and secret are derived and optional. Both nodes carry one drop-in (binary log, ROW, log_slave_updates, gtid_strict_mode, semi-sync both sides, TLS, the VIP bound beside the node's addresses through net.ipv6.ip_nonlocal_bind), so no role change restarts MariaDB.
Semi-sync. AFTER_SYNC, rpl_semi_sync_master_timeout=10000, rpl_semi_sync_master_wait_no_slave=OFF, slave_net_timeout=10; MariaDB waits for one replica's acknowledgement by design (wait_for_slave_count is MySQL's and 11.8 refuses to start with it: the first thing the end to end test caught). wait_no_slave=OFF is the third thing it caught: with both sides on and MariaDB's default, the replica's own SQL thread is a master with no slave and waited the 10 s on the first transaction it applied (traced: "Waiting for semi-sync ACK from slave" for 9.9 s); a node with no replica connected now does not wait, and a connected replica that does not answer still costs the primary the 10 s (the fallback). The fallback is database.server.semi_sync: drift in keel diff on the primary, and keel database watch (keel-database-watch.timer, 30 s) alerts once on the fallback and once on the recovery through the monitor's channels. A write in flight when a primary is cut off commits after the 10 s and is errant: such a primary comes back diverged (documented, tested in (d)'s second run).
The replica serves nothing. read_only at run time, and both nodes boot read only (the role's drop-in): follow lifts it on the holder at every boot, so a crashed old primary takes no write before it learns the newer epoch. READ_ONLY ADMIN revoked from every account but 'mysql'@'localhost' and mariadb.sys, root included (0049); keel's own statements go through runuser -u mysql -- mariadb, and a node's account and grant statements run with sql_log_bin off (a grant logged under the replica's server id would stop it under gtid_strict_mode, ER 1950). docs/apply.md's "root still writes" is corrected.
TLS. A database leaf per node from the mesh root CA (CN <member> mariadb, no etcd user; SANs the node's /128 and the VIP; serverAuth+clientAuth), the root made by keel mesh create in every cloud mode, asked of the holder or of the pair peer over the members' channel (issue, kind: database; the holder signs only for the address its spec gives the asker's key, and puts the VIP in the SANs only on the pair's signed record the request carries); the replication account REQUIRE SUBJECT '/CN=<peer> mariadb' AND ISSUER '/CN=<root>' (an etcd leaf of the same root is refused, measured: 1045); the server checks the root's CRL (ssl_crl), refreshed by watch with FLUSH SSL; MASTER_SSL_VERIFY_SERVER_CERT=1; renewed by watch. Files under /etc/mysql/keel-tls.
The credential is generated on the first primary, 0600 under /var/lib/keel/database/, asked by the other member over POST /v1/vip kind secret (answered only to the other member of the pair record): the first shared secret of 0041.
keel database follow (the role hook, keel-database-follow.path on /var/lib/keel/vip and at boot): holder → drained and promoted, read_only off, the privilege back; other member → read_only on, the lock, a fresh server seeded over TLS, a replica of the holder left alone, an old primary compared by GTID (@@gtid_binlog_state against the holder's state and gtid_slave_pos): none errant → dump kept, gtid_slave_pos = its gtid_binlog_pos, follows; some → stays read only, database.server.diverged in diff and keel database status, one alert; --destroy-local-database reseeds. keel database promote moves the VIP, then follows. keel database status; the role file /var/lib/keel/role for tracker#57.

Seams (for the end to end test)

Three, all the clients' or keel's own conventions, never switches of the code: KEEL_LIVE_ROOT names the tree that stands for a node's live system (keel.inspect.constants.ROOT_DEFAULT; never set on a machine); MYSQL_HOME names an options file with that node's socket, read by the MariaDB clients after /etc/mysql/my.cnf; KEEL_MARIADB_SERVICE names the node's unit (keel.system.dbmariadb.service()). The agents stand for keel-database-follow.path (follow on a change of the VIP's state) and keel-database-watch.timer (watch every 10 s there, 30 on a machine). The alert channel is a webhook the routing namespace serves.

Tests

Pure, 3300+ passing, coverage 97 % (gate 95): tests/test_system_dbpair.py (GTID comparison, the pair, the shared credential over the in-process channel and its refusals, the leaf issued by the holder and refused for another address or name, the pair plan, inspect's and diff's lines), tests/test_system_dbfollow.py (every branch of the follow table against a recorded MariaDB, watch's alerts once each way, status), tests/test_database_cli.py.

End to end, tests/test_mariadb_netns.py (CI job mariadb / trixie): tests/vip_netns.py's three nodes on 250 ms ±25 ms with 2 % loss measured first, etcd, the VIP's units, a MariaDB 11.8 server per node as a systemd unit in its namespace reading the node's root's drop-in directory. This PR tests case (a): A and B applied, B seeded over TLS (Master_SSL_Allowed: Yes, Using_Gtid: Slave_Pos), three writes at the VIP each read on B with the lag measured from the write's acknowledgement, semi-sync on and a commit's cost, the replica read only with 'mysql'@'localhost' alone holding READ_ONLY ADMIN, keel diff clean on both; then (the security review's three) B applied again and written through with its SQL thread still applying, B's etcd member leaf refused as repl where its database leaf is taken, and A restarted read only from its first second until follow lifts it.

Security review (2026-10-08), what changed: HIGH, the authorizations outside the binary log; MEDIUM, the VIP SAN bound to the signed pair record, REQUIRE SUBJECT ... AND ISSUER with the root's CRL, both nodes boot read only; LOW, the fence recorded before the address is dropped, the fallback alerted at its first sight and no recovery alert at the first watch, the bounds tightened to the measured figures. Not this PR: the rotation of the replication credential (an issue to open).

Deferred to the follow-up PR (branch feat/mariadb-pair-scenario, with the rest of the harness: a writer at the VIP, the partitions and heals, the alerts' webhook): (b) the fallback after 10 s with the replica cut, drift and the alert, writes going on, the recovery; (c) keel database promote while the application writes, the downtime, no acknowledged row lost; (d) the primary cut off idle, failover per keel#81, the database following, the old primary rejoining by itself; then cut off while written to, coming back diverged, read only, alerted, reseeded once confirmed; (e) the replica refusing INSERT as root and as the application. The harness for three servers on one runner took ten local iterations (all plumbing: a private /tmp for units in a netns, the servers' temporary directory, the manifests under the roots, the status parse), and found two product bugs on the way, both fixed here: MariaDB 11.8 has no rpl_semi_sync_master_wait_for_slave_count, and the certificate files were unreachable by the server as mysql (now /etc/mysql/keel-tls, 0644/0640).

Measured

Design case on every leg (measured over the overlay in CI: 226.5 ms minimum, 251 ms median, 3.75 % loss; locally 226.6 / 251 / 4 %).

What	Local, full harness (one host, 2026-10-08)	CI (`mariadb / trixie`)
(a) the replica's first connection to the primary (TLS over 250 ms), from its apply	5.3 s	5.3 s
(a) a commit at the VIP with the replica acknowledging (AFTER_SYNC)	0.28 to 0.31 s (one round trip)	0.30 to 0.33 s
(a) a client's write at the VIP (connect, TLS, commit)	1.6 to 1.7 s	1.3 to 1.9 s
(a) a write visible on the replica, from its acknowledgement	0.04 s (3 of 3)	0.05 to 0.13 s (3 of 3)
(a) the replica applied again, then written through	SQL thread applying, 0.04 s	SQL thread applying, 0.06 s
(a) B's etcd member leaf as `repl` / B's database leaf	1045 / taken	1045 / taken
(a) the primary restarted: read only at its first second; after `follow`	1 (0.8 s restart) / 0	1 (0.77 s restart) / 0
(a) `keel diff`, database section, both nodes	clean; `semi_sync: same (on)` on the primary	clean; `semi_sync: same (on)`
(b) first commit after the replica is cut; the next one	10.04 s; 0.04 s	follow-up
(b) fallback alerted; semi-sync back after the heal	at the next watch (immediately); 7.3 s	follow-up
(c) planned promote downtime, acknowledged rows on the new primary	not measured: the CLI refused the spec on the harness roots (no manifests), fixed since	follow-up
(d) failover, writes resume, auto-rejoin; diverged old primary	invalidated by (c)'s failure in that run	follow-up
(e) `INSERT` as root by the socket / as the application at the replica	refused, 1290 / 1290; only `'mysql'@'localhost'` holds `READ_ONLY ADMIN`	follow-up

Phase 2, step 3: MariaDB primary and replica over the mesh, semi-synchronous, the primary the VIP's holder (decisions 0013, 0020, 0028, 0031, 0049, 0050; tracker#57; the maintainer's answers of 2026-10-03 and 2026-10-11). The design is `docs/replication.md`; the four questions it asked were approved as recommended on 2026-10-11 (the precision to 0049 is Keel-Linux/handbook#57). keel 0.23.0. ## What it does - **Pair model.** `appliance.vip` plus `database.server.role`: the primary is the VIP's holder; `role` in the spec is the installation's, decides who seeds from whom before the first claim, and is runtime state from then on (0020). `inspect` reports the VIP's role; `diff` shows the difference as information, never drift; `apply` never converges it. With the pair record, `replication.primary`, `allowed_from` and `secret` are derived and optional. Both nodes carry one drop-in (binary log, ROW, `log_slave_updates`, `gtid_strict_mode`, semi-sync both sides, TLS, the VIP bound beside the node's addresses through `net.ipv6.ip_nonlocal_bind`), so no role change restarts MariaDB. - **Semi-sync.** `AFTER_SYNC`, `rpl_semi_sync_master_timeout=10000`, `rpl_semi_sync_master_wait_no_slave=OFF`, `slave_net_timeout=10`; MariaDB waits for one replica's acknowledgement by design (`wait_for_slave_count` is MySQL's and 11.8 refuses to start with it: the first thing the end to end test caught). `wait_no_slave=OFF` is the third thing it caught: with both sides on and MariaDB's default, the replica's own SQL thread is a master with no slave and waited the 10 s on the first transaction it applied (traced: "Waiting for semi-sync ACK from slave" for 9.9 s); a node with no replica connected now does not wait, and a connected replica that does not answer still costs the primary the 10 s (the fallback). The fallback is `database.server.semi_sync: drift` in `keel diff` on the primary, and `keel database watch` (`keel-database-watch.timer`, 30 s) alerts once on the fallback and once on the recovery through the monitor's channels. A write in flight when a primary is cut off commits after the 10 s and is errant: such a primary comes back diverged (documented, tested in (d)'s second run). - **The replica serves nothing.** `read_only` at run time, and **both nodes boot read only** (the role's drop-in): `follow` lifts it on the holder at every boot, so a crashed old primary takes no write before it learns the newer epoch. `READ_ONLY ADMIN` revoked from every account but `'mysql'@'localhost'` and `mariadb.sys`, **root included** (0049); keel's own statements go through `runuser -u mysql -- mariadb`, and a node's account and grant statements run with `sql_log_bin` off (a grant logged under the replica's server id would stop it under `gtid_strict_mode`, ER 1950). docs/apply.md's "root still writes" is corrected. - **TLS.** A `database` leaf per node from the mesh root CA (CN `<member> mariadb`, no etcd user; SANs the node's /128 and the VIP; serverAuth+clientAuth), the root made by `keel mesh create` in every cloud mode, asked of the holder or of the pair peer over the members' channel (`issue`, `kind: database`; the holder signs only for the address its spec gives the asker's key, and puts the VIP in the SANs only on the pair's signed record the request carries); the replication account `REQUIRE SUBJECT '/CN=<peer> mariadb' AND ISSUER '/CN=<root>'` (an etcd leaf of the same root is refused, measured: 1045); the server checks the root's CRL (`ssl_crl`), refreshed by watch with `FLUSH SSL`; `MASTER_SSL_VERIFY_SERVER_CERT=1`; renewed by watch. Files under `/etc/mysql/keel-tls`. - **The credential** is generated on the first primary, 0600 under `/var/lib/keel/database/`, asked by the other member over `POST /v1/vip` kind `secret` (answered only to the other member of the pair record): the first shared secret of 0041. - **`keel database follow`** (the role hook, `keel-database-follow.path` on `/var/lib/keel/vip` and at boot): holder → drained and promoted, `read_only` off, the privilege back; other member → `read_only` on, the lock, a fresh server seeded over TLS, a replica of the holder left alone, an old primary compared by GTID (`@@gtid_binlog_state` against the holder's state and `gtid_slave_pos`): none errant → dump kept, `gtid_slave_pos` = its `gtid_binlog_pos`, follows; some → stays read only, `database.server.diverged` in diff and `keel database status`, one alert; `--destroy-local-database` reseeds. `keel database promote` moves the VIP, then follows. `keel database status`; the role file `/var/lib/keel/role` for tracker#57. ## Seams (for the end to end test) Three, all the clients' or keel's own conventions, never switches of the code: `KEEL_LIVE_ROOT` names the tree that stands for a node's live system (`keel.inspect.constants.ROOT_DEFAULT`; never set on a machine); `MYSQL_HOME` names an options file with that node's socket, read by the MariaDB clients after `/etc/mysql/my.cnf`; `KEEL_MARIADB_SERVICE` names the node's unit (`keel.system.dbmariadb.service()`). The agents stand for `keel-database-follow.path` (follow on a change of the VIP's state) and `keel-database-watch.timer` (watch every 10 s there, 30 on a machine). The alert channel is a webhook the routing namespace serves. ## Tests Pure, 3300+ passing, coverage 97 % (gate 95): `tests/test_system_dbpair.py` (GTID comparison, the pair, the shared credential over the in-process channel and its refusals, the leaf issued by the holder and refused for another address or name, the pair plan, inspect's and diff's lines), `tests/test_system_dbfollow.py` (every branch of the follow table against a recorded MariaDB, watch's alerts once each way, status), `tests/test_database_cli.py`. End to end, `tests/test_mariadb_netns.py` (CI job `mariadb / trixie`): `tests/vip_netns.py`'s three nodes on 250 ms ±25 ms with 2 % loss measured first, etcd, the VIP's units, a MariaDB 11.8 server per node as a systemd unit in its namespace reading the node's root's drop-in directory. **This PR tests case (a)**: A and B applied, B seeded over TLS (`Master_SSL_Allowed: Yes`, `Using_Gtid: Slave_Pos`), three writes at the VIP each read on B with the lag measured from the write's acknowledgement, semi-sync on and a commit's cost, the replica read only with `'mysql'@'localhost'` alone holding `READ_ONLY ADMIN`, `keel diff` clean on both; then (the security review's three) B applied again and written through with its SQL thread still applying, B's etcd member leaf refused as `repl` where its database leaf is taken, and A restarted read only from its first second until `follow` lifts it. **Security review (2026-10-08), what changed**: HIGH, the authorizations outside the binary log; MEDIUM, the VIP SAN bound to the signed pair record, `REQUIRE SUBJECT ... AND ISSUER` with the root's CRL, both nodes boot read only; LOW, the fence recorded before the address is dropped, the fallback alerted at its first sight and no recovery alert at the first watch, the bounds tightened to the measured figures. Not this PR: the rotation of the replication credential (an issue to open). **Deferred to the follow-up PR** (branch `feat/mariadb-pair-scenario`, with the rest of the harness: a writer at the VIP, the partitions and heals, the alerts' webhook): (b) the fallback after 10 s with the replica cut, drift and the alert, writes going on, the recovery; (c) `keel database promote` while the application writes, the downtime, no acknowledged row lost; (d) the primary cut off idle, failover per keel#81, the database following, the old primary rejoining by itself; then cut off while written to, coming back diverged, read only, alerted, reseeded once confirmed; (e) the replica refusing `INSERT` as root and as the application. The harness for three servers on one runner took ten local iterations (all plumbing: a private `/tmp` for units in a netns, the servers' temporary directory, the manifests under the roots, the status parse), and found two product bugs on the way, both fixed here: MariaDB 11.8 has no `rpl_semi_sync_master_wait_for_slave_count`, and the certificate files were unreachable by the server as `mysql` (now `/etc/mysql/keel-tls`, 0644/0640). ## Measured Design case on every leg (measured over the overlay in CI: 226.5 ms minimum, 251 ms median, 3.75 % loss; locally 226.6 / 251 / 4 %). | What | Local, full harness (one host, 2026-10-08) | CI (`mariadb / trixie`) | | --- | --- | --- | | (a) the replica's first connection to the primary (TLS over 250 ms), from its apply | 5.3 s | 5.3 s | | (a) a commit at the VIP with the replica acknowledging (AFTER_SYNC) | 0.28 to 0.31 s (one round trip) | 0.30 to 0.33 s | | (a) a client's write at the VIP (connect, TLS, commit) | 1.6 to 1.7 s | 1.3 to 1.9 s | | (a) a write visible on the replica, from its acknowledgement | 0.04 s (3 of 3) | 0.05 to 0.13 s (3 of 3) | | (a) the replica applied again, then written through | SQL thread applying, 0.04 s | SQL thread applying, 0.06 s | | (a) B's etcd member leaf as `repl` / B's database leaf | 1045 / taken | 1045 / taken | | (a) the primary restarted: read only at its first second; after `follow` | 1 (0.8 s restart) / 0 | 1 (0.77 s restart) / 0 | | (a) `keel diff`, database section, both nodes | clean; `semi_sync: same (on)` on the primary | clean; `semi_sync: same (on)` | | (b) first commit after the replica is cut; the next one | 10.04 s; 0.04 s | follow-up | | (b) fallback alerted; semi-sync back after the heal | at the next watch (immediately); 7.3 s | follow-up | | (c) planned promote downtime, acknowledged rows on the new primary | not measured: the CLI refused the spec on the harness roots (no manifests), fixed since | follow-up | | (d) failover, writes resume, auto-rejoin; diverged old primary | invalidated by (c)'s failure in that run | follow-up | | (e) `INSERT` as root by the socket / as the application at the replica | refused, 1290 / 1290; only `'mysql'@'localhost'` holds `READ_ONLY ADMIN` | follow-up |

github-advanced-security[bot] (Migrated from github.com) reviewed

2026-10-08 12:31:09 -03:00

keel/system/dbtls.py

					
				@ -0,0 +187,4 @@

				        gid = grp.getgrnam(GROUP).gr_gid

				    except KeyError:

				        return

				    os.chmod(directory, DIR_MODE)

semgrep /

These permissions DIR_MODE are widely permissive and grant access to more people than may be necessary. A good default is 0o644 which gives read and write access to yourself and read access to everyone else.

Show more details

## semgrep / These permissions `DIR_MODE` are widely permissive and grant access to more people than may be necessary. A good default is `0o644` which gives read and write access to yourself and read access to everyone else. [Show more details](https://github.com/Keel-Linux/keel/security/code-scanning/28)

Sign in to join this conversation.

No reviewers