The appliance gate boots as many nodes as a caller declares #9

Merged

marcos-mendez merged 1 commit from ci/appliance-several-nodes into main

2026-09-27 07:46:12 -03:00

marcos-mendez commented

2026-09-27 07:41:29 -03:00

(Migrated from github.com)

What changes

test-appliance.yml takes a roles input, one role name per node. The count
is the number of names, so galera galera galera is three later and nothing
here assumes two; repeats are allowed because Galera nodes share a role, and
five is the ceiling because the runner is shared and a longer list is a typo
rather than a topology.

Empty, the default, is one container. Every new step is conditional on the
input, so no existing caller changes behaviour: keel-core, keel-nodebb,
keel-mariadb and keel-postgresql are called with the arguments they were
called with before.

The interface, and why

Declaring it: the roles input in the caller's own short workflow,
beside appliance and parent. The same kind of thing an appliance author
already writes there, and one line.
A node learning which one it is: /etc/keel/node.env in its own rootfs,
with KEEL_NODE_NAME, KEEL_NODE_ROLE, KEEL_NODE_INDEX and
KEEL_NODE_COUNT, and /etc/keel/peers.env once every node has an address.
A file in its own filesystem is how every other appliance setting arrives,
and unlike a hostname or a boot order it is the seam the appliance's own
replication feature takes over when the instance description carries the
role.
Addressing another node: by literal IPv6 address, never by name. On
Debian localhost resolves to IPv4 alone, so a name anywhere in this path
would quietly pick the wrong family. btn_tcp_probe_argv builds the bash
/dev/tcp connect one container runs against another, and the appliance's
own bt_wait_for turns it into a wait with a deadline.

Where the logic lives

lib/boot-test-nodes.sh, here rather than copied into four appliance
repositories, which is the trap docs/traps.md ends with and where the
handbook already decided shared boot test logic belongs. It has no side
effects, so it can be measured: 113 of 113 lines, 39 new bats tests, and
tests/coverage.sh now measures lib/ beside bin/. Total 164 of 164 lines,
100 percent.

Teardown

Runs whatever the outcome, cancellation included. It enumerates the containers
from LXC instead of recomputing them from the role list, so a node the test
renamed or started just before failing goes too. It stops all of them first
and removes the tree last, because keel-ci-cleanup stops one container and
deletes the tree in a single call: calling it once per node would delete the
configuration of the nodes not yet stopped and leave their init processes on a
rootfs that no longer exists. lxc-stop and lxc-destroy are already in the
runner's sudo policy, so several nodes need it no wider than one node did.

Test plan

actionlint 1.7.7, pinned by digest, clean on all five workflows
shellcheck -x lib/boot-test-nodes.sh clean
COVERAGE_THRESHOLD=100 tests/coverage.sh: 164/164, 100 percent
a real two node run of appliance / build-and-boot on keel-mariadb

Replication cannot be proved on one machine. Handbook decision 0013 says that if it cannot be tested in the gate then the replication work should not be built, because an untested replication feature in an appliance people trust with data is worse than no feature. This is that gate. Tracker issue Keel-Linux/tracker#4. ## What changes `test-appliance.yml` takes a `roles` input, one role name per node. The count is the number of names, so `galera galera galera` is three later and nothing here assumes two; repeats are allowed because Galera nodes share a role, and five is the ceiling because the runner is shared and a longer list is a typo rather than a topology. Empty, the default, is one container. Every new step is conditional on the input, so no existing caller changes behaviour: keel-core, keel-nodebb, keel-mariadb and keel-postgresql are called with the arguments they were called with before. ## The interface, and why - **Declaring it**: the `roles` input in the caller's own short workflow, beside `appliance` and `parent`. The same kind of thing an appliance author already writes there, and one line. - **A node learning which one it is**: `/etc/keel/node.env` in its own rootfs, with `KEEL_NODE_NAME`, `KEEL_NODE_ROLE`, `KEEL_NODE_INDEX` and `KEEL_NODE_COUNT`, and `/etc/keel/peers.env` once every node has an address. A file in its own filesystem is how every other appliance setting arrives, and unlike a hostname or a boot order it is the seam the appliance's own replication feature takes over when the instance description carries the role. - **Addressing another node**: by literal IPv6 address, never by name. On Debian `localhost` resolves to IPv4 alone, so a name anywhere in this path would quietly pick the wrong family. `btn_tcp_probe_argv` builds the bash `/dev/tcp` connect one container runs against another, and the appliance's own `bt_wait_for` turns it into a wait with a deadline. ## Where the logic lives `lib/boot-test-nodes.sh`, here rather than copied into four appliance repositories, which is the trap `docs/traps.md` ends with and where the handbook already decided shared boot test logic belongs. It has no side effects, so it can be measured: **113 of 113 lines, 39 new bats tests**, and `tests/coverage.sh` now measures `lib/` beside `bin/`. Total 164 of 164 lines, 100 percent. ## Teardown Runs whatever the outcome, cancellation included. It enumerates the containers from LXC instead of recomputing them from the role list, so a node the test renamed or started just before failing goes too. It stops all of them first and removes the tree last, because `keel-ci-cleanup` stops one container and deletes the tree in a single call: calling it once per node would delete the configuration of the nodes not yet stopped and leave their init processes on a rootfs that no longer exists. `lxc-stop` and `lxc-destroy` are already in the runner's sudo policy, so several nodes need it no wider than one node did. ## Test plan - [x] `actionlint` 1.7.7, pinned by digest, clean on all five workflows - [x] `shellcheck -x lib/boot-test-nodes.sh` clean - [x] `COVERAGE_THRESHOLD=100 tests/coverage.sh`: 164/164, 100 percent - [ ] a real two node run of `appliance / build-and-boot` on keel-mariadb