bt-layer leaves the build time systemctl and service wrappers in a child layer #6

New issue

Open

opened 2026-09-26 17:37:36 -03:00 by marcos-mendez · 1 comment

marcos-mendez commented

2026-09-26 17:37:36 -03:00

(Migrated from github.com)

bin/layer-lib adds turnkey.d/tkl-bashlib turnkey.d/systemd-chroot to a
child layer's build overlays (LAYER_CHILD_OVERLAYS), with the comment that
removelists-final/turnkey strips them from every finished layer and that a
child layer's conf scripts need them again. The adding happens; the stripping
of the finished child layer does not.

Result on the layers published on 2026-09-26:

core.rootfs/usr/local/bin:         apt-packages-origin gh_releases
nodejs-nginx.rootfs/usr/local/bin: apt-packages-origin gh_releases
nodebb.rootfs/usr/local/bin:       apt-packages-origin gh_releases service systemctl

Outside a build chroot those two wrappers call each other forever:
/usr/local/bin/systemctl stop ssh.service runs /usr/local/bin/service ssh stop, whose fallback runs systemctl again. A container assembled from the
nodebb layer therefore never finishes its first boot: it stops in
10regen-sshkeys with hundreds of alternating systemctl and service
processes, which is where the appliance gate
(appliance / build-and-boot in keel-nodebb) now fails.

Expected: a finished child layer is stripped with removelists-final/turnkey
the way a rootfs layer is, so /usr/local/bin/systemctl and
/usr/local/bin/service are not in the tarball.

Once fixed, nodebb (and any other child layer built since) has to be rebuilt
and republished to https://mirror.keellinux.org/layers before
appliance / build-and-boot can be made a required status on keel-nodebb's
main.

`bin/layer-lib` adds `turnkey.d/tkl-bashlib turnkey.d/systemd-chroot` to a child layer's build overlays (`LAYER_CHILD_OVERLAYS`), with the comment that `removelists-final/turnkey` strips them from every finished layer and that a child layer's conf scripts need them again. The adding happens; the stripping of the finished child layer does not. Result on the layers published on 2026-09-26: core.rootfs/usr/local/bin: apt-packages-origin gh_releases nodejs-nginx.rootfs/usr/local/bin: apt-packages-origin gh_releases nodebb.rootfs/usr/local/bin: apt-packages-origin gh_releases service systemctl Outside a build chroot those two wrappers call each other forever: `/usr/local/bin/systemctl stop ssh.service` runs `/usr/local/bin/service ssh stop`, whose fallback runs `systemctl` again. A container assembled from the `nodebb` layer therefore never finishes its first boot: it stops in `10regen-sshkeys` with hundreds of alternating `systemctl` and `service` processes, which is where the appliance gate (`appliance / build-and-boot` in keel-nodebb) now fails. Expected: a finished child layer is stripped with `removelists-final/turnkey` the way a rootfs layer is, so `/usr/local/bin/systemctl` and `/usr/local/bin/service` are not in the tarball. Once fixed, `nodebb` (and any other child layer built since) has to be rebuilt and republished to `https://mirror.keellinux.org/layers` before `appliance / build-and-boot` can be made a required status on keel-nodebb's `main`.

marcos-mendez commented

2026-09-26 18:15:25 -03:00

(Migrated from github.com)

Correcting the diagnosis: the root cause was found on the keel-nodebb side and
is narrower than what this issue says.

The overlay ships /etc/confconsole/services.txt, a conffile of confconsole, so
the explicit upgrade of inithooks and confconsole in the recipe met a dpkg
conffile prompt. Conf scripts run with stdin closed, dpkg read end of file, and
root.patched failed, which is why fab never got to remove the build overlays.
Fixed in keel-linux/keel-nodebb#1.

What remains here, and is the reason to keep this open: bt-layer packed and
exported that tree anyway. make failed and the layer was still written,
published and served, so a broken layer reached the mirror with a valid manifest
and checksum. A layer whose build did not finish should not be packed, and the
appliance gate should not be the first thing to notice.

Correcting the diagnosis: the root cause was found on the keel-nodebb side and is narrower than what this issue says. The overlay ships `/etc/confconsole/services.txt`, a conffile of confconsole, so the explicit upgrade of inithooks and confconsole in the recipe met a dpkg conffile prompt. Conf scripts run with stdin closed, dpkg read end of file, and `root.patched` failed, which is why fab never got to remove the build overlays. Fixed in keel-linux/keel-nodebb#1. What remains here, and is the reason to keep this open: `bt-layer` packed and exported that tree anyway. `make` failed and the layer was still written, published and served, so a broken layer reached the mirror with a valid manifest and checksum. A layer whose build did not finish should not be packed, and the appliance gate should not be the first thing to notice.