ci: boot the published layer without root on keel-lxc-1 #16

Merged

marcos-mendez merged 1 commit from ci/unprivileged-boot-test into main

2026-10-01 01:29:31 -03:00

marcos-mendez commented

2026-10-01 01:20:43 -03:00

(Migrated from github.com)

Why

The runner keel-lxc-1 reached root through sudo: apt-get, the lxc-* commands and the two root helpers keel-ci-boot-test and keel-ci-cleanup. Each of those is root-equivalent (install any .deb, a custom LXC config mounting host paths, lxc-attach), so any job on the runner could become root on the public services VM that also serves keellinux.org, archive, mirror and releases.

What changes

bin/unprivileged-lxc run SCRATCH -- tests/boot-test.sh ... runs the repository's own boot test, unchanged, as root in a user namespace (lxc-usernsexec) mapped onto the runner's subordinate ids (runner:165536:65536), which are also the ids the container runs as. The assembled rootfs therefore has the owners the container sees. The runner's uid is mapped in at 65536, so the workspace and scratch tree stay writable.
Root in that namespace cannot add a veth to lxcbr0 or enter a container's cgroup. The test's lxc-* commands are links to the same script that forward to a broker outside the namespace, running as the runner:
- lxc-start gets userns.conf, the lxc.idmap and the AppArmor profile lxc-container-default-with-nesting on top of the config the test wrote, inside systemd-run --user --scope -p Delegate=yes;
- lxc-attach runs in a scope of the same user manager and gets the stdin the test piped to it (keel-mariadb pipes SQL).
- The broker has the runner's rights and no more.
tar is a link too, because a user namespace may not create device nodes. It forgives exactly that refusal, and fails if anything else is mixed in. LXC mounts its own /dev over the rootfs anyway.
unprivileged-lxc cleanup removes the subuid-owned scratch tree through the same namespace. The cleanup step no longer uses sudo.
None of the nine tests/boot-test.sh in the appliance repositories needs to change. The job id and the check name appliance / boot-published-layer are unchanged.
build-deb.yml still calls sudo apt-get. Nothing calls it, and profile/WORKFLOWS.md now says it has to move into an unprivileged container first.

The host side (subuid, lxc-usernet, the runner's LXC default config, linger, removal of /etc/sudoers.d/runner and the root helpers) is recorded in docs/ci-cd.md section 6 of the keel repository.

Evidence

tests/unprivileged-lxc.bats, 22 cases. tests/coverage.sh locally: 100 percent of bin/unprivileged-lxc (124/124), total 100.
By hand on keel-lxc-1 as runner, without sudo: keel-core's tests/boot-test.sh against the published core layer passed in 27 s. The assemble skipped 9 device nodes, the container had a ULA address in 6 s, first boot finished 5 s later, keel diff reported 6 same and 0 drift, and cleanup removed the tree.
keel-core's appliance / boot-published-layer against this branch (tooling_ref set to it, temporary caller branch since deleted), on keel-lxc-1:
- with the runner's sudo still in place but unused: https://github.com/Keel-Linux/keel-core/actions/runs/36814690796, green in 44 s;
- after /etc/sudoers.d/runner and both root helpers were removed: https://github.com/Keel-Linux/keel-core/actions/runs/36814955977, green in 44 s, the container up with an address in 6 s, first boot finished 5 s later, keel diff 6 same 0 drift, 0 monitors left.

Test plan

lint / actionlint and tests / coverage green
keel-core appliance check green against this branch, with the runner's sudo removed
after merge, one appliance PR shows appliance / boot-published-layer green from main

## Why The runner `keel-lxc-1` reached root through sudo: `apt-get`, the `lxc-*` commands and the two root helpers `keel-ci-boot-test` and `keel-ci-cleanup`. Each of those is root-equivalent (install any .deb, a custom LXC config mounting host paths, `lxc-attach`), so any job on the runner could become root on the public services VM that also serves keellinux.org, archive, mirror and releases. ## What changes - `bin/unprivileged-lxc run SCRATCH -- tests/boot-test.sh ...` runs the repository's own boot test, unchanged, as root in a user namespace (`lxc-usernsexec`) mapped onto the runner's subordinate ids (`runner:165536:65536`), which are also the ids the container runs as. The assembled rootfs therefore has the owners the container sees. The runner's uid is mapped in at 65536, so the workspace and scratch tree stay writable. - Root in that namespace cannot add a veth to `lxcbr0` or enter a container's cgroup. The test's `lxc-*` commands are links to the same script that forward to a broker outside the namespace, running as the runner: - `lxc-start` gets `userns.conf`, the `lxc.idmap` and the AppArmor profile `lxc-container-default-with-nesting` on top of the config the test wrote, inside `systemd-run --user --scope -p Delegate=yes`; - `lxc-attach` runs in a scope of the same user manager and gets the stdin the test piped to it (keel-mariadb pipes SQL). - The broker has the runner's rights and no more. - `tar` is a link too, because a user namespace may not create device nodes. It forgives exactly that refusal, and fails if anything else is mixed in. LXC mounts its own `/dev` over the rootfs anyway. - `unprivileged-lxc cleanup` removes the subuid-owned scratch tree through the same namespace. The cleanup step no longer uses sudo. - None of the nine `tests/boot-test.sh` in the appliance repositories needs to change. The job id and the check name `appliance / boot-published-layer` are unchanged. - `build-deb.yml` still calls `sudo apt-get`. Nothing calls it, and `profile/WORKFLOWS.md` now says it has to move into an unprivileged container first. The host side (subuid, `lxc-usernet`, the runner's LXC default config, linger, removal of `/etc/sudoers.d/runner` and the root helpers) is recorded in docs/ci-cd.md section 6 of the keel repository. ## Evidence - `tests/unprivileged-lxc.bats`, 22 cases. `tests/coverage.sh` locally: 100 percent of `bin/unprivileged-lxc` (124/124), total 100. - By hand on keel-lxc-1 as `runner`, without sudo: keel-core's `tests/boot-test.sh` against the published `core` layer passed in 27 s. The assemble skipped 9 device nodes, the container had a ULA address in 6 s, first boot finished 5 s later, `keel diff` reported 6 same and 0 drift, and cleanup removed the tree. - keel-core's `appliance / boot-published-layer` against this branch (`tooling_ref` set to it, temporary caller branch since deleted), on keel-lxc-1: - with the runner's sudo still in place but unused: https://github.com/Keel-Linux/keel-core/actions/runs/36814690796, green in 44 s; - after `/etc/sudoers.d/runner` and both root helpers were removed: https://github.com/Keel-Linux/keel-core/actions/runs/36814955977, green in 44 s, the container up with an address in 6 s, first boot finished 5 s later, `keel diff` 6 same 0 drift, 0 monitors left. ## Test plan - [ ] `lint / actionlint` and `tests / coverage` green - [x] keel-core appliance check green against this branch, with the runner's sudo removed - [ ] after merge, one appliance PR shows `appliance / boot-published-layer` green from main