Left after validating the 2026-09-26 layer: a failed build was published, and the declarative boot has no path of its own #2

New issue

Open

opened 2026-09-26 15:24:14 -03:00 by marcos-mendez · 1 comment

marcos-mendez commented

2026-09-26 15:24:14 -03:00

(Migrated from github.com)

Found while validating the nodebb layer of 2026-09-26 (sha256 40e3f4da,
parent nodejs-nginx f791777f) in a second container, forum2, on the
build host. Recorded in docs/forum-build-2026-09-26.md. Pull request #1
fixes the two that belong to this recipe; everything below is what is left.

1. The published layer is the output of a failed build, and must be replaced

conf.d/main met a dpkg conffile prompt on /etc/confconsole/services.txt
and root.patched failed with make: *** Error 100. The recipe half is
fixed in #1. Two consequences survive in the artifact that is on the build
host right now and mirrored nowhere yet:

confconsole is install ok unpacked, not configured.
root.patched never finished, so fab never removed the build overlays:
the layer carries /usr/local/bin/systemctl and /usr/local/bin/service,
the systemd-chroot shims. core.rootfs and nodejs-nginx.rootfs do not
have them. At the first boot 10regen-sshkeys called the shim and looped
forever.

The layer has to be rebuilt with the recipe of #1 and the old file replaced
everywhere it has been copied. Until then nothing should be assembled from
it without removing the two shims by hand.

2. bt-layer publishes a build that failed

This is the reason #1 above was not caught at build time. bt-layer line
155:

make build/stamps/root.patched "${make_args[@]}" </dev/null 2>&1 | tee -a "$log" || true

The || true discards the status, and the script goes on to copy
root.patched, pack the tarball, write the manifest and export the layer.
A build that fails is indistinguishable from one that succeeds unless
somebody reads the log. It should stop on a non-zero make, and the exit
status has to be taken from PIPESTATUS because of the tee.

This is a change in the buildtasks fork, not in this repository, but it is
recorded here because this is where it did its damage.

3. 00declarative and the keel package disagree on where the spec lives

00declarative reads $INITHOOKS_DECL, whose packaged default in
/etc/default/inithooks is /etc/inithooks.yaml. The keel package, the
example in keel/instance.example.yaml and keel spec apply all use
/etc/keel/instance.yaml. An image with a spec in the documented place
therefore boots with 00declarative doing nothing, silently, and falls
through to 29preseed generating a random password instead.

forum2 was booted with INITHOOKS_DECL pointed at the keel path, which is
the only reason the declarative boot could be demonstrated at all. One of
the two packages should move: either inithooks defaults to
/etc/keel/instance.yaml, or the keel package ships the symlink or the
default file entry. Brief section 11 reserves the final spec path for the
maintainer, so this needs a decision before either package changes.

4. A Keel container has no inithooks unit that runs at boot

build-tag proxmox writes /var/lib/turnkey-info/inithooks.service/lxc,
and the headless patch's inithooks.service carries
ConditionPathExists=!/var/lib/turnkey-info/inithooks.service/lxc. No
inithooks-lxc.service exists in this image, so nothing runs the first boot
on lxc-start: forum2 needed lxc-attach -- /usr/lib/inithooks/run, and
the earlier forum needed the same plus keel spec apply.

For an appliance whose whole point is that a spec file boots it, that is the
missing piece. Either the Keel container patch ships an
inithooks-lxc.service that runs the chain headless, or it stops writing
the marker and relies on the headless unit.

5. 29preseed always exits 1

Its first line is [[ -n "$_TURNKEY_INIT" ]] && exit 0 under bash -e. A
&& list whose own status is non-zero ends the script, so the hook exits 1
whenever _TURNKEY_INIT is unset, which is every non-interactive boot. It
is a harmless no-op (it would have exited 0 a line later anyway), but every
boot log carries ERR: [29preseed] failed - exit code 1, which trains the
reader to ignore errors in that log. Upstream file, shipped by the
buildtasks headless patch.

ls /usr/lib/confconsole/plugins.d/Instance exits 2. The plugins are the
five upstream ones; keelcli is not in the image either. The changelog of
confconsole 2.2.3+keel1 is the IPv6 networking work and the tests, and
mentions no Instance plugin.

Brief section 6 asks for Instance: view, edit, apply spec, show drift,
export spec, each with a headless equivalent. The library calls it would sit
on (keel spec apply, keel diff, keel inspect) all exist and work in the
image. This is not a regression, it is unwritten work, but the build report
assumed it was in the layer, so it is worth having on the list.

Not in this issue

The four drifts keel diff reports (network.ipv6.method, tls.acme
declared while disabled, security.updates skip against force,
instance.fqdn unknown) reproduced exactly on forum2 and are already
recorded in the coordinator's section of docs/forum-build-2026-09-26.md.
They belong to the keel repository.

Found while validating the nodebb layer of 2026-09-26 (sha256 `40e3f4da`, parent nodejs-nginx `f791777f`) in a second container, `forum2`, on the build host. Recorded in docs/forum-build-2026-09-26.md. Pull request #1 fixes the two that belong to this recipe; everything below is what is left. ## 1. The published layer is the output of a failed build, and must be replaced `conf.d/main` met a dpkg conffile prompt on `/etc/confconsole/services.txt` and `root.patched` failed with `make: *** Error 100`. The recipe half is fixed in #1. Two consequences survive in the artifact that is on the build host right now and mirrored nowhere yet: - confconsole is `install ok unpacked`, not configured. - `root.patched` never finished, so fab never removed the build overlays: the layer carries `/usr/local/bin/systemctl` and `/usr/local/bin/service`, the systemd-chroot shims. `core.rootfs` and `nodejs-nginx.rootfs` do not have them. At the first boot `10regen-sshkeys` called the shim and looped forever. The layer has to be rebuilt with the recipe of #1 and the old file replaced everywhere it has been copied. Until then nothing should be assembled from it without removing the two shims by hand. ## 2. bt-layer publishes a build that failed This is the reason #1 above was not caught at build time. `bt-layer` line 155: ``` make build/stamps/root.patched "${make_args[@]}" </dev/null 2>&1 | tee -a "$log" || true ``` The `|| true` discards the status, and the script goes on to copy `root.patched`, pack the tarball, write the manifest and export the layer. A build that fails is indistinguishable from one that succeeds unless somebody reads the log. It should stop on a non-zero make, and the exit status has to be taken from `PIPESTATUS` because of the `tee`. This is a change in the buildtasks fork, not in this repository, but it is recorded here because this is where it did its damage. ## 3. 00declarative and the keel package disagree on where the spec lives `00declarative` reads `$INITHOOKS_DECL`, whose packaged default in `/etc/default/inithooks` is `/etc/inithooks.yaml`. The keel package, the example in `keel/instance.example.yaml` and `keel spec apply` all use `/etc/keel/instance.yaml`. An image with a spec in the documented place therefore boots with `00declarative` doing nothing, silently, and falls through to `29preseed` generating a random password instead. `forum2` was booted with `INITHOOKS_DECL` pointed at the keel path, which is the only reason the declarative boot could be demonstrated at all. One of the two packages should move: either inithooks defaults to `/etc/keel/instance.yaml`, or the keel package ships the symlink or the default file entry. Brief section 11 reserves the final spec path for the maintainer, so this needs a decision before either package changes. ## 4. A Keel container has no inithooks unit that runs at boot `build-tag proxmox` writes `/var/lib/turnkey-info/inithooks.service/lxc`, and the headless patch's `inithooks.service` carries `ConditionPathExists=!/var/lib/turnkey-info/inithooks.service/lxc`. No `inithooks-lxc.service` exists in this image, so nothing runs the first boot on `lxc-start`: `forum2` needed `lxc-attach -- /usr/lib/inithooks/run`, and the earlier `forum` needed the same plus `keel spec apply`. For an appliance whose whole point is that a spec file boots it, that is the missing piece. Either the Keel container patch ships an `inithooks-lxc.service` that runs the chain headless, or it stops writing the marker and relies on the headless unit. ## 5. 29preseed always exits 1 Its first line is `[[ -n "$_TURNKEY_INIT" ]] && exit 0` under `bash -e`. A `&&` list whose own status is non-zero ends the script, so the hook exits 1 whenever `_TURNKEY_INIT` is unset, which is every non-interactive boot. It is a harmless no-op (it would have exited 0 a line later anyway), but every boot log carries `ERR: [29preseed] failed - exit code 1`, which trains the reader to ignore errors in that log. Upstream file, shipped by the buildtasks headless patch. ## 6. The confconsole Instance menu does not exist `ls /usr/lib/confconsole/plugins.d/Instance` exits 2. The plugins are the five upstream ones; `keelcli` is not in the image either. The changelog of `confconsole 2.2.3+keel1` is the IPv6 networking work and the tests, and mentions no Instance plugin. Brief section 6 asks for **Instance**: view, edit, apply spec, show drift, export spec, each with a headless equivalent. The library calls it would sit on (`keel spec apply`, `keel diff`, `keel inspect`) all exist and work in the image. This is not a regression, it is unwritten work, but the build report assumed it was in the layer, so it is worth having on the list. ## Not in this issue The four drifts `keel diff` reports (`network.ipv6.method`, `tls.acme` declared while disabled, `security.updates` skip against force, `instance.fqdn` unknown) reproduced exactly on `forum2` and are already recorded in the coordinator's section of docs/forum-build-2026-09-26.md. They belong to the keel repository.

marcos-mendez commented

2026-09-26 15:28:48 -03:00

(Migrated from github.com)

7. The appliance check cannot run on a public repository

Found while merging #1. tests / build-and-boot targets [self-hosted, keel-lxc]. The runner keel-lxc-1 is online, idle and carries that label, but the organization's Default runner group has allows_public_repositories: false and every repository in the organization is public, so the job is never offered to it. It stays queued and GitHub cancels it after 24 hours. That is the state today: no appliance test has ever run in CI, and a pull request that waits for that check waits forever.

GitHub refuses this combination on purpose: a self-hosted runner on a public repository will execute whatever a fork's pull request contains. Turning the flag on is a security decision, not a configuration detail, so it is left to the maintainer. The options, in the order they cost:

Keep the flag off and make the appliance test a manual step on the build host, quoted in the pull request, which is what decision 0003 already allows and what happened for #1.
Keep the flag off and run the appliance test on a schedule from a private repository that checks this one out by tag.
Turn the flag on and restrict the runner group to named repositories, with pull_request_target avoided everywhere and no workflow that runs fork code before approval.

Independently of the flag, the job would still fail: the runner has no fab and no buildtasks, so bt-layer is not there (docs/releases-host.md section 7 already records that test-appliance.yml has to fetch layers from mirror.keellinux.org instead of building them).

## 7. The appliance check cannot run on a public repository Found while merging #1. `tests / build-and-boot` targets `[self-hosted, keel-lxc]`. The runner `keel-lxc-1` is online, idle and carries that label, but the organization's Default runner group has `allows_public_repositories: false` and every repository in the organization is public, so the job is never offered to it. It stays queued and GitHub cancels it after 24 hours. That is the state today: no appliance test has ever run in CI, and a pull request that waits for that check waits forever. GitHub refuses this combination on purpose: a self-hosted runner on a public repository will execute whatever a fork's pull request contains. Turning the flag on is a security decision, not a configuration detail, so it is left to the maintainer. The options, in the order they cost: 1. Keep the flag off and make the appliance test a manual step on the build host, quoted in the pull request, which is what decision 0003 already allows and what happened for #1. 2. Keep the flag off and run the appliance test on a schedule from a private repository that checks this one out by tag. 3. Turn the flag on and restrict the runner group to named repositories, with `pull_request_target` avoided everywhere and no workflow that runs fork code before approval. Independently of the flag, the job would still fail: the runner has no fab and no buildtasks, so `bt-layer` is not there (docs/releases-host.md section 7 already records that `test-appliance.yml` has to fetch layers from `mirror.keellinux.org` instead of building them).