Two node boot test in the gate, before replication ships #4

New issue

Closed

opened 2026-09-27 06:12:22 -03:00 by marcos-mendez · 1 comment

marcos-mendez commented

2026-09-27 06:12:22 -03:00

(Migrated from github.com)

Today test-appliance.yml in Keel-Linux/.github pulls a published layer,
assembles it, boots one container on the self hosted runner and runs the
repository's tests/boot-test.sh. What is needed is the same for two
containers on the same bridge, with each one's role given to it, and a way for
the test to wait for the second to be reachable from the first over IPv6
without either of them being configured by hand.

Closes when a gate run boots two containers from the same published layer,
configures one as primary and one as replica, writes on the primary, reads the
same row on the replica, and tears both down whatever the outcome. Galera later
needs three, so the shape must not assume exactly two.

Touches Keel-Linux/.github (the reusable workflow), the runner's sudo policy
on the public services VM, and each database appliance's boot test.

Depends on nothing. Blocks the console modes on all three database engines.

The appliance gate boots one container. Replication cannot be proved that way, and handbook decision 0013 says plainly that if this cannot be tested in the gate then the replication work should not be built: an untested replication feature in an appliance people trust with data is worse than no feature. Today `test-appliance.yml` in `Keel-Linux/.github` pulls a published layer, assembles it, boots one container on the self hosted runner and runs the repository's `tests/boot-test.sh`. What is needed is the same for two containers on the same bridge, with each one's role given to it, and a way for the test to wait for the second to be reachable from the first over IPv6 without either of them being configured by hand. Closes when a gate run boots two containers from the same published layer, configures one as primary and one as replica, writes on the primary, reads the same row on the replica, and tears both down whatever the outcome. Galera later needs three, so the shape must not assume exactly two. Touches `Keel-Linux/.github` (the reusable workflow), the runner's sudo policy on the public services VM, and each database appliance's boot test. Depends on nothing. Blocks the console modes on all three database engines.

marcos-mendez commented

2026-09-27 08:31:02 -03:00

(Migrated from github.com)

Done. The gate boots as many containers as a caller declares, and replication
between two of them is proved in it.

Keel-Linux/.github #9, merged: the roles input on test-appliance.yml
and lib/boot-test-nodes.sh, the shared topology logic, at 100 percent
(164/164 lines, 39 new bats tests). tests/coverage.sh there now measures
lib/ beside bin/.
Keel-Linux/keel-mariadb #7, merged: roles: primary replica in the caller,
the boot test running its whole check set on every node, and the replication
phase. Its library stays at 100 percent (248/248, 24 new tests).
Handbook: STATUS.md, "The gate boots two nodes, and replication is proved in
it (2026-09-27)".

The interface

roles, a space separated list of role names, one per node, in the caller's
own workflow beside appliance and parent. The count is the number of names,
so galera galera galera is three for phase 4 and nothing assumes two;
repeats are allowed because Galera nodes share a role.

A node learns which one it is from /etc/keel/node.env in its own rootfs, and
gets /etc/keel/peers.env with every node's literal IPv6 address once the
bridge has handed them out. A file in its own filesystem, which is how every
other appliance setting arrives, and the seam the console modes take over when
the instance description carries the role. The test addresses another node by
literal IPv6 address and never by a name, because on Debian a name resolves to
IPv4 alone.

The run

keel-mariadb run 36314204371, appliance / build-and-boot 1m18s: two
containers from the published mariadb layer (0adca434 on core
7acf2c53), assembled in 16s and 14s, both addressed from lxcbr0 6s after
start, both first boots done 16s after start. On each node: the declared
password authenticated, webmin-mysql installed, Webmin 200, keel diff 6
same and 0 drift. Then each node's 3306 answered from the other over IPv6,
Slave_running ON, and the row written on the primary read back from the
replica from the primary container by an account whose password the
description declared.

What is hand configuration

All of the phase, deliberately. The table in keel-mariadb's tests/README.md
puts each piece next to the console screen of 0013 that will own it: the role
file against the instance description of phase 2; the MariaDB drop-in
(server_id, the bind address gaining this node's own literal global address,
log_bin on the primary alone) against the Primary and Replica screens; the
replication account against the Primary screen; CHANGE MASTER TO ... MASTER_USE_GTID=slave_pos against the Replica screen. Accounts are authorised
for the peer's /64 rather than one address, which is what 0013 already recorded
about credentials between nodes.

The sudo policy needed no widening

keel-ci-cleanup takes one container and one tree and does both in one call,
so calling it per node would delete the configuration of the nodes not stopped
yet. The teardown therefore stops every container LXC reports in the run's own
lxcpath with lxc-stop and lxc-destroy, which are already in
/etc/sudoers.d/runner, and then makes the one keel-ci-cleanup call that
removes the tree. Nothing on the public services VM was changed.

Teardown runs on a cancelled run too: the boot test traps INT and TERM as
well as EXIT, and the workflow step is if: always(). It reports how many
container monitors of the run are still alive; both runs reported 0.

Cancellation, measured rather than asserted

On a throwaway branch, run 36315031422 was cancelled with both containers up
and the boot test mid check. The always() step stopped and destroyed both,
keel-ci-cleanup removed the tree, and 0 container monitors were left, four
seconds after the cancellation; watched from outside the job at the same time,
2 monitors and 1 tree at 11:14:09, 0 monitors at 11:14:25, 0 trees at 11:14:30.

That check corrected something worth knowing: the boot test's own traps do
not fire on a cancelled job. It runs as root through sudo keel-ci-boot-test, and the unprivileged runner user cannot signal a root
child, so the workflow step is what does the work. The traps still matter for a
run started by hand, where the signal reaches the process.

One note for whoever picks up phase 3 for PostgreSQL: the digest in this issue,
d5220c56, is the postgresql layer. The published mariadb layer is
0adca434 on the same parent core 7acf2c53, and that is what was used.
PostgreSQL's own two node phase is the next step and is still waiting on its
rebuilt layer being published.

Done. The gate boots as many containers as a caller declares, and replication between two of them is proved in it. - `Keel-Linux/.github` #9, merged: the `roles` input on `test-appliance.yml` and `lib/boot-test-nodes.sh`, the shared topology logic, at 100 percent (164/164 lines, 39 new bats tests). `tests/coverage.sh` there now measures `lib/` beside `bin/`. - `Keel-Linux/keel-mariadb` #7, merged: `roles: primary replica` in the caller, the boot test running its whole check set on every node, and the replication phase. Its library stays at 100 percent (248/248, 24 new tests). - Handbook: STATUS.md, "The gate boots two nodes, and replication is proved in it (2026-09-27)". ### The interface `roles`, a space separated list of role names, one per node, in the caller's own workflow beside `appliance` and `parent`. The count is the number of names, so `galera galera galera` is three for phase 4 and nothing assumes two; repeats are allowed because Galera nodes share a role. A node learns which one it is from `/etc/keel/node.env` in its own rootfs, and gets `/etc/keel/peers.env` with every node's literal IPv6 address once the bridge has handed them out. A file in its own filesystem, which is how every other appliance setting arrives, and the seam the console modes take over when the instance description carries the role. The test addresses another node by literal IPv6 address and never by a name, because on Debian a name resolves to IPv4 alone. ### The run keel-mariadb run 36314204371, `appliance / build-and-boot` **1m18s**: two containers from the published `mariadb` layer (`0adca434` on `core` `7acf2c53`), assembled in 16s and 14s, both addressed from `lxcbr0` 6s after start, both first boots done 16s after start. On each node: the declared password authenticated, `webmin-mysql` installed, Webmin 200, `keel diff` 6 same and 0 drift. Then each node's 3306 answered from the other over IPv6, `Slave_running ON`, and the row written on the primary read back from the replica **from the primary container** by an account whose password the description declared. ### What is hand configuration All of the phase, deliberately. The table in keel-mariadb's `tests/README.md` puts each piece next to the console screen of 0013 that will own it: the role file against the instance description of phase 2; the MariaDB drop-in (`server_id`, the bind address gaining this node's own literal global address, `log_bin` on the primary alone) against the Primary and Replica screens; the replication account against the Primary screen; `CHANGE MASTER TO ... MASTER_USE_GTID=slave_pos` against the Replica screen. Accounts are authorised for the peer's /64 rather than one address, which is what 0013 already recorded about credentials between nodes. ### The sudo policy needed no widening `keel-ci-cleanup` takes one container and one tree and does both in one call, so calling it per node would delete the configuration of the nodes not stopped yet. The teardown therefore stops every container LXC reports in the run's own lxcpath with `lxc-stop` and `lxc-destroy`, which are already in `/etc/sudoers.d/runner`, and then makes the one `keel-ci-cleanup` call that removes the tree. Nothing on the public services VM was changed. Teardown runs on a cancelled run too: the boot test traps `INT` and `TERM` as well as `EXIT`, and the workflow step is `if: always()`. It reports how many container monitors of the run are still alive; both runs reported 0. ### Cancellation, measured rather than asserted On a throwaway branch, run 36315031422 was cancelled with both containers up and the boot test mid check. The `always()` step stopped and destroyed both, `keel-ci-cleanup` removed the tree, and 0 container monitors were left, four seconds after the cancellation; watched from outside the job at the same time, 2 monitors and 1 tree at 11:14:09, 0 monitors at 11:14:25, 0 trees at 11:14:30. That check corrected something worth knowing: **the boot test's own traps do not fire on a cancelled job.** It runs as root through `sudo keel-ci-boot-test`, and the unprivileged `runner` user cannot signal a root child, so the workflow step is what does the work. The traps still matter for a run started by hand, where the signal reaches the process. One note for whoever picks up phase 3 for PostgreSQL: the digest in this issue, `d5220c56`, is the **postgresql** layer. The published `mariadb` layer is `0adca434` on the same parent `core` `7acf2c53`, and that is what was used. PostgreSQL's own two node phase is the next step and is still waiting on its rebuilt layer being published.