ci: upgrade the published layer from trixie-testing before its first boot #20

Merged

marcos-mendez merged 1 commit from ci/boot-upgraded-layer into main

2026-10-03 11:26:28 -03:00

marcos-mendez commented

2026-10-03 10:44:43 -03:00

(Migrated from github.com)

Fixes Keel-Linux/tracker#53, option (a).

What changes

The first time a boot test starts a container, the broker in bin/unprivileged-lxc runs an upgrade boot before the real one:

It boots the container once, with the first boot masked on the init command line: systemd.mask= for inithooks, keel-host-keys, turnkey-init-fence and systemd-machine-id-commit. Before anything else runs, it checks that these units report masked-runtime.
It runs bin/upgrade-rootfs inside the container, through lxc-attach … bash -s.
It stops the container, then starts it as the test asked.

Inside the container, bin/upgrade-rootfs:

runs apt-get update --error-on=any and full-upgrade. Sources are the image's own keel.sources with trixie-testing on, plus Debian trixie, trixie-updates and trixie-security.
stops services from starting: policy-rc.d returns 101 during the upgrade.
installs the .deb files of the new package_artifact input and checks that each one is installed at its own version.

The versions that differ from the published layer go to the job log and the job summary before the first boot. None of the nine boot tests needs a change.

Security

This addresses the review of the first version:

Maintainer scripts run only in the container. They run under the container's own idmap, which does not map the runner, and under the AppArmor profile of the real boot. Nothing of the upgrade runs on the runner, and there is no chroot and no fresh /proc.
Every cleanup step runs inside the container, so no host-side path can follow a symlink planted by a package.
keel.sources is refused if any stanza names a host other than https://archive.keellinux.org, has no Signed-By, or is trusted or insecure.
No resolution is overridden. The gate's containers cannot reach archive.keellinux.org: the public IPv4 address refuses them and the IPv6 address drops them (measured 2026-10-03; tracker#5 recorded them reaching it on 2026-09-30). Instead, the runner fetches the signed indexes of trixie and trixie-testing and the packages they name, once per run, over https only. Pool paths in the indexes are validated before anything is fetched. The files are copied into each container, and apt reads them through a file: copy of keel.sources that keeps its Signed-By, so the signature and every hash are still checked inside the container.
Network access: the runner contacts archive.keellinux.org and the container contacts deb.debian.org. Nothing else.
Unchanged: the runner group, refs/heads/main, and the refusal of fork PRs.
Checkouts: both checkouts of the job now set persist-credentials: false.

For callers

An artifact is visible only inside the run that uploaded it. A repo that wants its own .deb in the boot moves its appliance job into packages.yml, with needs: build and package_artifact: <name>. The check keeps its name. This is a follow-up in each repo.

Evidence

keel-core throwaway branch, test-appliance.yml@main with tooling_ref: ci/boot-upgraded-layer at 3206594: run 37129396959 passed. During the upgrade boot it moved confconsole 2.2.3+keel14 -> +keel15, inithooks 2.3.6+keel21 -> +keel23 and keel 0.15.3 -> 0.15.4. Then the real first boot finished and the core boot test passed. The branch has been deleted.
package_artifact cannot be exercised through main's workflow. It is covered by the bats cases, which check the copy into the container, the install arguments and the version check.
tests / coverage is at 100 percent on every file, and actionlint 1.7.7 reports nothing.

Fixes Keel-Linux/tracker#53, option (a). ## What changes The first time a boot test starts a container, the broker in `bin/unprivileged-lxc` runs an upgrade boot before the real one: 1. It boots the container once, with the first boot masked on the init command line: `systemd.mask=` for inithooks, keel-host-keys, turnkey-init-fence and systemd-machine-id-commit. Before anything else runs, it checks that these units report `masked-runtime`. 2. It runs `bin/upgrade-rootfs` inside the container, through `lxc-attach … bash -s`. 3. It stops the container, then starts it as the test asked. Inside the container, `bin/upgrade-rootfs`: - runs `apt-get update --error-on=any` and `full-upgrade`. Sources are the image's own `keel.sources` with trixie-testing on, plus Debian trixie, trixie-updates and trixie-security. - stops services from starting: `policy-rc.d` returns 101 during the upgrade. - installs the `.deb` files of the new `package_artifact` input and checks that each one is installed at its own version. The versions that differ from the published layer go to the job log and the job summary before the first boot. None of the nine boot tests needs a change. ## Security This addresses the review of the first version: - **Maintainer scripts run only in the container.** They run under the container's own idmap, which does not map the runner, and under the AppArmor profile of the real boot. Nothing of the upgrade runs on the runner, and there is no chroot and no fresh `/proc`. - **Every cleanup step runs inside the container**, so no host-side path can follow a symlink planted by a package. - **`keel.sources` is refused** if any stanza names a host other than https://archive.keellinux.org, has no `Signed-By`, or is trusted or insecure. - **No resolution is overridden.** The gate's containers cannot reach archive.keellinux.org: the public IPv4 address refuses them and the IPv6 address drops them (measured 2026-10-03; tracker#5 recorded them reaching it on 2026-09-30). Instead, the runner fetches the signed indexes of trixie and trixie-testing and the packages they name, once per run, over https only. Pool paths in the indexes are validated before anything is fetched. The files are copied into each container, and apt reads them through a `file:` copy of `keel.sources` that keeps its `Signed-By`, so the signature and every hash are still checked inside the container. - **Network access:** the runner contacts archive.keellinux.org and the container contacts deb.debian.org. Nothing else. - **Unchanged:** the runner group, `refs/heads/main`, and the refusal of fork PRs. - **Checkouts:** both checkouts of the job now set `persist-credentials: false`. ## For callers An artifact is visible only inside the run that uploaded it. A repo that wants its own `.deb` in the boot moves its `appliance` job into `packages.yml`, with `needs: build` and `package_artifact: <name>`. The check keeps its name. This is a follow-up in each repo. ## Evidence - keel-core throwaway branch, `test-appliance.yml@main` with `tooling_ref: ci/boot-upgraded-layer` at 3206594: run 37129396959 passed. During the upgrade boot it moved confconsole 2.2.3+keel14 -> +keel15, inithooks 2.3.6+keel21 -> +keel23 and keel 0.15.3 -> 0.15.4. Then the real first boot finished and the core boot test passed. The branch has been deleted. - `package_artifact` cannot be exercised through main's workflow. It is covered by the bats cases, which check the copy into the container, the install arguments and the version check. - `tests / coverage` is at 100 percent on every file, and actionlint 1.7.7 reports nothing.