Every machine makes its own keys before a service reads one #19

Merged

marcos-mendez merged 2 commits from fix/generate-host-keys into master

2026-09-29 01:33:13 -03:00

marcos-mendez commented

2026-09-29 01:10:06 -03:00

(Migrated from github.com)

Problem

keel-core#8: every layer and all fourteen templates on mirror.keellinux.org carry the same SSH host keys, the same turnkey-ssl TLS key in /etc/ssl/private and, in lapp and postgresql, the same snakeoil key. A machine only got its own from 10regen-sshkeys and 15regen-sslcert; one where they did not run served a key anyone can download, and nothing said so. BRIEF section 5.3: every appliance gets its own host keys.

Fix

keel-host-keys.service, ordered before sshd, Webmin, the web servers, PostgreSQL, postfix and the init fence, runs bin/keel-host-keys on every boot. It generates any SSH host key, TLS pair or snakeoil pair (where ssl-cert is installed) that is missing.
It replaces a key in use whose public half is in shared-host-keys, measured from every layer and template on the mirror. That is what fixes machines already made from the published images. Replacements are logged at warning to the auth facility.
A key it cannot make, or that is still shared afterwards, fails the unit.
15regen-sslcert restarts Webmin too; it kept serving the previous key until a reboot.

Evidence

tests/test-host-keys.bats, 20 tests, real ssh-keygen: bin/keel-host-keys 64/64, shell total 99.65. 4 of 5 mutations turn it red; the survivor (dropping the ssh/tls kind comparison) is equivalent, since the two digest formats cannot collide.
Run against the keys extracted from the published core.tar.zst: all three SSH host keys and the TLS key recognised and replaced.

Deploying the three together

This is one of three pull requests for keel-core#8: Keel-Linux/inithooks (generate on the machine), Keel-Linux/common (remove from the build), Keel-Linux/buildtasks (refuse at export). Merging deploys nothing: the build host checkouts are updated by hand. The order that matters is deployment:

Publish inithooks 2.3.6+keel6 to the archive (attended, signing). Until then a layer without keys would boot with no TLS certificate until 15regen-sslcert, which only restarts services already running.
Update /turnkey/fab-keel/common and /turnkey/buildtasks-keel on the build host in the same sitting: the audit without the removals refuses every layer.
Rebuild the chain and publish (attended). Images published before this still carry the keys; keel-host-keys replaces them on any machine made from one.

## Problem keel-core#8: every layer and all fourteen templates on mirror.keellinux.org carry the same SSH host keys, the same turnkey-ssl TLS key in `/etc/ssl/private` and, in lapp and postgresql, the same snakeoil key. A machine only got its own from `10regen-sshkeys` and `15regen-sslcert`; one where they did not run served a key anyone can download, and nothing said so. BRIEF section 5.3: every appliance gets its own host keys. ## Fix - `keel-host-keys.service`, ordered before sshd, Webmin, the web servers, PostgreSQL, postfix and the init fence, runs `bin/keel-host-keys` on every boot. It generates any SSH host key, TLS pair or snakeoil pair (where ssl-cert is installed) that is missing. - It replaces a key in use whose public half is in `shared-host-keys`, measured from every layer and template on the mirror. That is what fixes machines already made from the published images. Replacements are logged at warning to the auth facility. - A key it cannot make, or that is still shared afterwards, fails the unit. - `15regen-sslcert` restarts Webmin too; it kept serving the previous key until a reboot. ## Evidence - `tests/test-host-keys.bats`, 20 tests, real ssh-keygen: `bin/keel-host-keys` 64/64, shell total 99.65. 4 of 5 mutations turn it red; the survivor (dropping the ssh/tls kind comparison) is equivalent, since the two digest formats cannot collide. - Run against the keys extracted from the published `core.tar.zst`: all three SSH host keys and the TLS key recognised and replaced. ## Deploying the three together This is one of three pull requests for keel-core#8: Keel-Linux/inithooks (generate on the machine), Keel-Linux/common (remove from the build), Keel-Linux/buildtasks (refuse at export). Merging deploys nothing: the build host checkouts are updated by hand. The order that matters is deployment: 1. Publish `inithooks 2.3.6+keel6` to the archive (attended, signing). Until then a layer without keys would boot with no TLS certificate until `15regen-sslcert`, which only restarts services already running. 2. Update `/turnkey/fab-keel/common` and `/turnkey/buildtasks-keel` on the build host in the same sitting: the audit without the removals refuses every layer. 3. Rebuild the chain and publish (attended). Images published before this still carry the keys; `keel-host-keys` replaces them on any machine made from one.