One first boot run at a time, and keel-init as the command #25

Merged

marcos-mendez merged 3 commits from fix/one-first-boot-run into master

2026-09-30 02:21:27 -03:00

marcos-mendez commented

2026-09-30 01:31:11 -03:00

(Migrated from github.com)

Fixes #24. Fixes #22.

One first boot run at a time (#24)

On keel-wordpress 19.0-3 (LXC on Proxmox, 2026-09-30), the boot run (/usr/lib/inithooks/run, pid 260) was waiting in 30rootpass on the container console (pts/1). A turnkey-init typed after pct enter started a second run of the same hooks. It set the root password and then hung in 35mysqlpass.

The fix is a lock that both runs hold:

An exclusive flock(2) on /run/inithooks.lock (override with INITHOOKS_LOCK), held for as long as hooks run. The holder writes its pid into the file, so a refusal can name it.
keel-init does not wait. While another run holds the lock it exits 75 and says where that run is waiting: pct console <ctid> on Proxmox, or lxc-console -n <name> on LXC, or the VM's console (it picks the hint with systemd-detect-virt). The container id cannot be known from inside, so the hint says <ctid>.
run waits for the lock instead, because systemd cannot come back later. It takes the lock before it reads /etc/default/inithooks, so after a keel-init that completed the first boot (98finalize sets RUN_FIRSTBOOT=false) it skips the firstboot hooks.
A lock cannot outlive its run. The kernel drops the lock when the run exits, however it exits. run closes the descriptor for each hook ({INIT_LOCK_FD}>&-), and keel-init's descriptor is not inheritable (PEP 446), so a daemon left behind by a hook does not hold the lock. Both runs release the lock before confconsole starts, because confconsole stays open.
/etc/update-motd.d/06-keel-init prints keel-init --status at login. While a run is in progress it says where to answer the wizard. While the fence is up and no run is in progress it says to run keel-init. Otherwise it prints nothing. The login message and the refusal use the same text from one function.

After review (second commit)

keel-init no longer takes over the first boot. inithooks.service starts after getty.target, so keel-init could get the lock before the boot run. It now also refuses (exit 75) while RUN_FIRSTBOOT=true and run has not yet written /run/inithooks-complete. It makes this check while holding the lock. A test covers the race with the real run: keel-init goes first and is refused, the boot run then does the first boot and writes the marker, and keel-init runs after that.
The message names the right place. The lock file now describes whoever holds it (kind, pid, tty, dtach, phase, hook, preseeded), and run rewrites it before every hook. The refusal changes with the holder:
- a keel-init running in dtach: dtach -a <socket>
- a keel-init on a terminal: the name of that terminal
- a preseeded boot run, or the everyboot phase: wait for it to finish
- an interactive boot run: the console hint
Two small fixes. keel-init now tries the lock for 0.5 s before refusing, so the login message's check of the lock cannot make it refuse. There is also a new no-manual-page lintian override for usr/sbin/keel-init.

Second review (third commit)

A first boot that nothing runs. When the first boot is pending, keel-init now checks the state of inithooks.service:
- If the unit is active, activating or deactivating, or has a start job queued, keel-init refuses as before.
- Otherwise nothing will ever run the first boot: the unit was stopped, failed, or was skipped by its container condition. keel-init then execs /usr/lib/inithooks/run on the operator's terminal.
Why run the boot run rather than only print the remedy: in the container case systemctl start inithooks is skipped again by the same condition, so the only remedy that works is running run by hand, and keel-init can do that itself. Why not keel-init's own hooks: they are a reduced set (_TURNKEY_INIT, blacklist) and would do the incomplete first boot the previous commit prevents. run takes the lock itself, so if the unit starts in the meantime, it waits and then finds the first boot done.

Tests: a stopped unit, a failed unit, a skipped unit, and a boot run killed in its wizard. In each case the real run then does the first boot.
Two smaller fixes. RUN_FIRSTBOOT is now read by bash sourcing the file, exactly as run reads it, so export, quotes and trailing comments are handled the same way. A relative dtach socket is now resolved against the dtach process's working directory, so the attach hint works from any shell.
tests/test_vocabulary.py: the two local failures are a test-environment issue, not a bug. The default KEEL_SRC points at a sibling keel checkout that predates keel#48. With KEEL_SRC set to current keel main, master and this branch both pass, and CI skips these tests.

keel-init as the command (#22, handbook decision 0015)

keel-init is the real file. turnkey-init is a relative link to it (turnkey-init -> keel-init), committed in the tree and installed by dh_install as a link. I checked this by building the package in a trixie container: the .deb contains ./usr/sbin/turnkey-init -> keel-init.
Help and messages name the command the operator typed. --help and --status do not need root.
The README and the fence page now say keel-init. The image turnkey-init-root.png keeps its file name.

Tests

tests/test_init_lock.py (38 tests): the library; keel-init loaded in-process; keel-init and turnkey-init run as processes; the link checked as relative; and the reported case, with the real run blocked in a 30rootpass hook while keel-init refuses, names run's pid and does not run the hook. After the boot run finishes, keel-init runs.
tests/test-init-lock.bats (10 tests), six new tests in tests/test-run.bats, and tests/test-motd.bats (4 tests). Every lock is a real flock, and the other holder is a real process.
Mutation checks: I removed, one at a time, the lock in run, the release before confconsole, the per-hook close, and keel-init's lock or its release. Each one made the suite fail.
kcov: lib/init-lock.sh 21/21, shell total 99.71 (gate 98). Python: libinithooks/init_lock.py 100 percent, total 99 (gate 95). The details are in COVERAGE.md.

Follow-ups in other repositories (not changed here)

common: overlays/turnkey.d/turnkey-init-fence/root/.profile.d/turnkey-init-fence starts turnkey-init in dtach at first login. It should start keel-init. conf/turnkey.d/motd writes 07-check-inithooks and 08-turnkey-confconsole; neither names turnkey-init.
buildtasks: patches/headless/overlay/root/.profile.d/turnkey-init-fence and patches/container/overlay/root/.profile.d/turnkey-init-fence run turnkey-init --full-confconsole. They should run keel-init.
keel: firstboot.d/10keel-system (comment), tests/hook.bats (test name) and docs/apply.md name turnkey-init.
handbook: in decision 0015's table, the /usr/sbin/turnkey-init row can say that this package does it now.
confconsole and keel-core: grep finds no text that tells an operator to run turnkey-init, so nothing to change there.
Still turnkey-* in this package (0015's table): turnkey-sudoadmin, turnkey-install-security-updates, turnkey-init-fence.

Fixes #24. Fixes #22. ## One first boot run at a time (#24) On keel-wordpress 19.0-3 (LXC on Proxmox, 2026-09-30), the boot run (`/usr/lib/inithooks/run`, pid 260) was waiting in `30rootpass` on the container console (pts/1). A `turnkey-init` typed after `pct enter` started a second run of the same hooks. It set the root password and then hung in `35mysqlpass`. The fix is a lock that both runs hold: - An exclusive `flock(2)` on `/run/inithooks.lock` (override with `INITHOOKS_LOCK`), held for as long as hooks run. The holder writes its pid into the file, so a refusal can name it. - `keel-init` does not wait. While another run holds the lock it exits 75 and says where that run is waiting: `pct console <ctid>` on Proxmox, or `lxc-console -n <name>` on LXC, or the VM's console (it picks the hint with `systemd-detect-virt`). The container id cannot be known from inside, so the hint says `<ctid>`. - `run` waits for the lock instead, because systemd cannot come back later. It takes the lock before it reads `/etc/default/inithooks`, so after a keel-init that completed the first boot (98finalize sets `RUN_FIRSTBOOT=false`) it skips the firstboot hooks. - A lock cannot outlive its run. The kernel drops the lock when the run exits, however it exits. `run` closes the descriptor for each hook (`{INIT_LOCK_FD}>&-`), and keel-init's descriptor is not inheritable (PEP 446), so a daemon left behind by a hook does not hold the lock. Both runs release the lock before confconsole starts, because confconsole stays open. - `/etc/update-motd.d/06-keel-init` prints `keel-init --status` at login. While a run is in progress it says where to answer the wizard. While the fence is up and no run is in progress it says to run `keel-init`. Otherwise it prints nothing. The login message and the refusal use the same text from one function. ### After review (second commit) - **keel-init no longer takes over the first boot.** `inithooks.service` starts after `getty.target`, so `keel-init` could get the lock before the boot run. It now also refuses (exit 75) while `RUN_FIRSTBOOT=true` and `run` has not yet written `/run/inithooks-complete`. It makes this check while holding the lock. A test covers the race with the real `run`: keel-init goes first and is refused, the boot run then does the first boot and writes the marker, and keel-init runs after that. - **The message names the right place.** The lock file now describes whoever holds it (`kind`, `pid`, `tty`, `dtach`, `phase`, `hook`, `preseeded`), and `run` rewrites it before every hook. The refusal changes with the holder: - a keel-init running in dtach: `dtach -a <socket>` - a keel-init on a terminal: the name of that terminal - a preseeded boot run, or the everyboot phase: wait for it to finish - an interactive boot run: the console hint - **Two small fixes.** keel-init now tries the lock for 0.5 s before refusing, so the login message's check of the lock cannot make it refuse. There is also a new `no-manual-page` lintian override for `usr/sbin/keel-init`. ### Second review (third commit) - **A first boot that nothing runs.** When the first boot is pending, keel-init now checks the state of `inithooks.service`: - If the unit is active, activating or deactivating, or has a start job queued, keel-init refuses as before. - Otherwise nothing will ever run the first boot: the unit was stopped, failed, or was skipped by its container condition. keel-init then execs `/usr/lib/inithooks/run` on the operator's terminal. **Why run the boot run rather than only print the remedy:** in the container case `systemctl start inithooks` is skipped again by the same condition, so the only remedy that works is running `run` by hand, and keel-init can do that itself. Why not keel-init's own hooks: they are a reduced set (`_TURNKEY_INIT`, blacklist) and would do the incomplete first boot the previous commit prevents. `run` takes the lock itself, so if the unit starts in the meantime, it waits and then finds the first boot done. Tests: a stopped unit, a failed unit, a skipped unit, and a boot run killed in its wizard. In each case the real `run` then does the first boot. - **Two smaller fixes.** `RUN_FIRSTBOOT` is now read by bash sourcing the file, exactly as `run` reads it, so `export`, quotes and trailing comments are handled the same way. A relative dtach socket is now resolved against the dtach process's working directory, so the attach hint works from any shell. - **`tests/test_vocabulary.py`:** the two local failures are a test-environment issue, not a bug. The default `KEEL_SRC` points at a sibling `keel` checkout that predates keel#48. With `KEEL_SRC` set to current keel main, master and this branch both pass, and CI skips these tests. ## keel-init as the command (#22, handbook decision 0015) - `keel-init` is the real file. `turnkey-init` is a relative link to it (`turnkey-init -> keel-init`), committed in the tree and installed by `dh_install` as a link. I checked this by building the package in a trixie container: the `.deb` contains `./usr/sbin/turnkey-init -> keel-init`. - Help and messages name the command the operator typed. `--help` and `--status` do not need root. - The README and the fence page now say `keel-init`. The image `turnkey-init-root.png` keeps its file name. ## Tests - `tests/test_init_lock.py` (38 tests): the library; keel-init loaded in-process; `keel-init` and `turnkey-init` run as processes; the link checked as relative; and the reported case, with the real `run` blocked in a `30rootpass` hook while keel-init refuses, names run's pid and does not run the hook. After the boot run finishes, keel-init runs. - `tests/test-init-lock.bats` (10 tests), six new tests in `tests/test-run.bats`, and `tests/test-motd.bats` (4 tests). Every lock is a real flock, and the other holder is a real process. - Mutation checks: I removed, one at a time, the lock in `run`, the release before confconsole, the per-hook close, and keel-init's lock or its release. Each one made the suite fail. - kcov: `lib/init-lock.sh` 21/21, shell total 99.71 (gate 98). Python: `libinithooks/init_lock.py` 100 percent, total 99 (gate 95). The details are in COVERAGE.md. ## Follow-ups in other repositories (not changed here) - **common**: `overlays/turnkey.d/turnkey-init-fence/root/.profile.d/turnkey-init-fence` starts `turnkey-init` in dtach at first login. It should start `keel-init`. `conf/turnkey.d/motd` writes `07-check-inithooks` and `08-turnkey-confconsole`; neither names turnkey-init. - **buildtasks**: `patches/headless/overlay/root/.profile.d/turnkey-init-fence` and `patches/container/overlay/root/.profile.d/turnkey-init-fence` run `turnkey-init --full-confconsole`. They should run `keel-init`. - **keel**: `firstboot.d/10keel-system` (comment), `tests/hook.bats` (test name) and `docs/apply.md` name `turnkey-init`. - **handbook**: in decision 0015's table, the `/usr/sbin/turnkey-init` row can say that this package does it now. - **confconsole** and **keel-core**: `grep` finds no text that tells an operator to run `turnkey-init`, so nothing to change there. - Still `turnkey-*` in this package (0015's table): `turnkey-sudoadmin`, `turnkey-install-security-updates`, `turnkey-init-fence`.