layer: export every layer with an empty machine-id #15

Merged

marcos-mendez merged 1 commit from fix/empty-machine-id into 19.x

2026-09-30 07:31:13 -03:00

marcos-mendez commented

2026-09-30 07:25:41 -03:00

(Migrated from github.com)

Problem

Every Keel image ships the same /etc/machine-id, f0e97605ab594989b4d78f4a126b5b37 (templateB2: mode 0444, plus /var/lib/dbus/machine-id as a regular file with the same id). It comes from the fab bootstrap (/turnkey/fab-keel/bootstraps/trixie-amd64/etc/machine-id, dated 2026-09-22). systemd keeps a populated file, and systemd's postinst runs systemd-machine-id-setup on every configure, so a child layer that upgrades systemd writes one of its own. Every LXC or VM made from one image then shares its DHCP DUID/IAID, journal and networkd identity, and keel's MariaDB server_id: a replica test found two containers with the same one. This is the defect in docs/traps.md, "Every appliance built from core has the same machine-id".

Fix

bt-layer runs the new layer_reset_machine_id on root.patched after layer_check_build. That is before the rootfs copy that children start from and before the tarball, so every layer gets it, whatever its build did:

/etc/machine-id becomes an empty file (0444). Debian's cloud images ship it this way (config_space/scripts/LAST/10-cleanup), and machine-id(5) recommends it. The file is not left missing or set to uninitialized, because either one makes the first boot a systemd first boot, and that presets every unit.
If /var/lib/dbus/machine-id exists, it becomes a link to /etc/machine-id, which is what tmpfiles.d/dbus.conf makes. When /etc/machine-id is empty, systemd falls back to the D-Bus file, so a regular D-Bus file would bring the old id back.
A tree already in that state is not written, so a child does not copy its parent's empty file into its own tarball. If the reset fails, the export stops.

TurnKey does the same in neither buildtasks (patches/container or bt-container) nor bootstrap, and its tracker has no issue about it.

Evidence

tests/layer: tests were written first and failed before the fix. They cover each case of the function, the core tarball and rootfs of a build that wrote an id, and a bt-layer run whose id cannot be reset. Each of five mutations turns the suite red. Coverage: layer-lib 383/384, bt-layer 111/112, the same one uncovered line each as on 19.x.
Real run on the test VM:
- On an overlay deck, a child that wrote an id ends with an empty 0444 file and the link in its upper directory. One that did not write an id leaves its upper directory empty.
- The templateB2 rootfs as published boots with f0e976.... Reset with this function and booted from scratch twice in LXC, it came up with 0308a979... and then 0bbc049e.... Each id was kept across a reboot, the D-Bus link followed it, no first boot was detected, and the same 41 units were enabled as in the published image.

Deploying

Merging deploys nothing. /turnkey/buildtasks-keel on the build host has to be updated by hand before the next build. Images and machines that already exist keep the shared id. To fix a machine, run rm -f /etc/machine-id /var/lib/dbus/machine-id && systemd-machine-id-setup && ln -s /etc/machine-id /var/lib/dbus/machine-id, then reboot and run keel apply again. This changes that node's server_id.

## Problem Every Keel image ships the same `/etc/machine-id`, `f0e97605ab594989b4d78f4a126b5b37` (templateB2: mode 0444, plus `/var/lib/dbus/machine-id` as a regular file with the same id). It comes from the fab bootstrap (`/turnkey/fab-keel/bootstraps/trixie-amd64/etc/machine-id`, dated 2026-09-22). systemd keeps a populated file, and systemd's postinst runs `systemd-machine-id-setup` on every configure, so a child layer that upgrades systemd writes one of its own. Every LXC or VM made from one image then shares its DHCP DUID/IAID, journal and networkd identity, and keel's MariaDB `server_id`: a replica test found two containers with the same one. This is the defect in docs/traps.md, "Every appliance built from core has the same machine-id". ## Fix `bt-layer` runs the new `layer_reset_machine_id` on `root.patched` after `layer_check_build`. That is before the rootfs copy that children start from and before the tarball, so every layer gets it, whatever its build did: - `/etc/machine-id` becomes an empty file (0444). Debian's cloud images ship it this way (`config_space/scripts/LAST/10-cleanup`), and machine-id(5) recommends it. The file is not left missing or set to `uninitialized`, because either one makes the first boot a systemd first boot, and that presets every unit. - If `/var/lib/dbus/machine-id` exists, it becomes a link to `/etc/machine-id`, which is what `tmpfiles.d/dbus.conf` makes. When `/etc/machine-id` is empty, systemd falls back to the D-Bus file, so a regular D-Bus file would bring the old id back. - A tree already in that state is not written, so a child does not copy its parent's empty file into its own tarball. If the reset fails, the export stops. TurnKey does the same in neither buildtasks (`patches/container` or `bt-container`) nor bootstrap, and its tracker has no issue about it. ## Evidence - `tests/layer`: tests were written first and failed before the fix. They cover each case of the function, the `core` tarball and rootfs of a build that wrote an id, and a `bt-layer` run whose id cannot be reset. Each of five mutations turns the suite red. Coverage: layer-lib 383/384, bt-layer 111/112, the same one uncovered line each as on 19.x. - Real run on the test VM: - On an overlay deck, a child that wrote an id ends with an empty 0444 file and the link in its upper directory. One that did not write an id leaves its upper directory empty. - The templateB2 rootfs as published boots with `f0e976...`. Reset with this function and booted from scratch twice in LXC, it came up with `0308a979...` and then `0bbc049e...`. Each id was kept across a reboot, the D-Bus link followed it, no first boot was detected, and the same 41 units were enabled as in the published image. ## Deploying Merging deploys nothing. `/turnkey/buildtasks-keel` on the build host has to be updated by hand before the next build. Images and machines that already exist keep the shared id. To fix a machine, run `rm -f /etc/machine-id /var/lib/dbus/machine-id && systemd-machine-id-setup && ln -s /etc/machine-id /var/lib/dbus/machine-id`, then reboot and run `keel apply` again. This changes that node's `server_id`.