A shared private key ships inside the layer #8

New issue

Open

opened 2026-09-28 00:23:30 -03:00 by marcos-mendez · 2 comments

marcos-mendez commented

2026-09-28 00:23:30 -03:00

(Migrated from github.com)

The turnkey-ssl package ships a private key inside the .deb, identical on
every image built from this layer.

Verified on the wordpress-demo container, 2026-09-28:

dpkg -S /usr/share/turnkey-ssl/cert.key  ->  turnkey-ssl
-rw-r--r-- root root 1704  /usr/share/turnkey-ssl/cert.key
subject=O=TurnKey GNU/Linux, OU=Software appliances, CN=tkldev
notAfter=Aug 19 05:44:20 2025 GMT

The key and the certificate match (same modulus digest), the file is
world-readable, and the certificate expired over a year ago.

Not currently exploited on this appliance. 15regen-sslcert did run and
the key in service is a different one: /etc/ssl/private/cert.key, mode
0400, CN=wordpress-demo, valid to 2036. The shipped key is dead weight
here.

Why it is still a defect. docs/traps.md records 15regen-sslcert as one
of the hooks that fails when a container blocks on its own console, alongside
95secupdates. So there is a plausible path to an appliance serving TLS with a
private key that is published in a package and a certificate that expired in
2025 — and nothing detects it. The failure is silent: the service starts, the
port answers, and only the certificate tells you.

What closes this

No private key ships in any Keel layer. A key is generated on the machine or
it does not exist.
An appliance that reaches a login prompt with the shared key still in place
must say so, loudly. A first-boot hook that failed is exactly the case where
nobody is reading the log.
Decide what replaces turnkey-ssl. Keel-Linux/tracker#15 recommends
rebuilding it as ours rather than upstreaming it, and this is the reason why:
a package that ships a private key cannot go to Debian.
The stale /usr/share/turnkey-ssl/cert.key should not survive on disk after
regeneration.

Org-wide context: Keel-Linux/tracker#15.

The `turnkey-ssl` package ships a **private key** inside the .deb, identical on every image built from this layer. Verified on the `wordpress-demo` container, 2026-09-28: dpkg -S /usr/share/turnkey-ssl/cert.key -> turnkey-ssl -rw-r--r-- root root 1704 /usr/share/turnkey-ssl/cert.key subject=O=TurnKey GNU/Linux, OU=Software appliances, CN=tkldev notAfter=Aug 19 05:44:20 2025 GMT The key and the certificate match (same modulus digest), the file is world-readable, and the certificate expired over a year ago. **Not currently exploited on this appliance.** `15regen-sslcert` did run and the key in service is a different one: `/etc/ssl/private/cert.key`, mode `0400`, `CN=wordpress-demo`, valid to 2036. The shipped key is dead weight here. **Why it is still a defect.** `docs/traps.md` records `15regen-sslcert` as one of the hooks that fails when a container blocks on its own console, alongside `95secupdates`. So there is a plausible path to an appliance serving TLS with a private key that is published in a package and a certificate that expired in 2025 — and nothing detects it. The failure is silent: the service starts, the port answers, and only the certificate tells you. ## What closes this - No private key ships in any Keel layer. A key is generated on the machine or it does not exist. - An appliance that reaches a login prompt with the shared key still in place must say so, loudly. A first-boot hook that failed is exactly the case where nobody is reading the log. - Decide what replaces `turnkey-ssl`. Keel-Linux/tracker#15 recommends rebuilding it as ours rather than upstreaming it, and this is the reason why: a package that ships a private key cannot go to Debian. - The stale `/usr/share/turnkey-ssl/cert.key` should not survive on disk after regeneration. Org-wide context: Keel-Linux/tracker#15.

marcos-mendez commented

2026-09-29 00:22:57 -03:00

(Migrated from github.com)

Measured on the build host on 2026-09-29, read only, across all twelve layer rootfs trees in /mnt/builds/layers: the shared TLS key is not the only one, and it is not only in /usr/share.

File	Where it comes from	Same in every layer
`/etc/ssl/private/cert.key`, `cert.pem`	`turnkey-ssl` postinst copies the shipped key here when none exists	yes, public key sha256 `139a0fd7...`
`/usr/share/turnkey-ssl/cert.key`, `cert.pem`	the package payload, mode 0644	yes, the same key
`/etc/ssh/ssh_host_{rsa,ecdsa,ed25519}_key`	openssh-server postinst at build time	yes, ed25519 `SHA256:co8r09MoEC8FVPxSBL5ltte5n7uA3dvi8FeWS05+mUQ`
`/etc/webmin/miniserv.pem`	webmin postinst at build time; unused, `miniserv.conf` points at `cert.pem`	yes
`/etc/ssl/private/ssl-cert-snakeoil.key`	ssl-cert, in `lapp` and `postgresql` only	differs per layer, same in every image of that layer

So every image built from any published template starts with the same SSH host keys and the same TLS key in the place services read, until 10regen-sshkeys and 15regen-sslcert replace them. If either hook does not run, the machine serves a key anyone can download.

The one public appliance is not affected: forum serves its own key on 443 and 12321 (0ce6f58b...) and has its own SSH host key.

The fix, in two parts:

Nothing ships. common/removelists-final/turnkey removes all of the above as the last step of every layer build, and a dpkg path-exclude stops turnkey-ssl from putting its key back on upgrade.
Generated before anything reads it. A oneshot unit ordered before ssh, webmin, the web servers, postgresql and the init fence generates any key that is missing, on the machine. A service can then never start with a shared key or with none, whether the first boot hooks ran or not.

Measured on the build host on 2026-09-29, read only, across all twelve layer rootfs trees in `/mnt/builds/layers`: the shared TLS key is not the only one, and it is not only in `/usr/share`. | File | Where it comes from | Same in every layer | | --- | --- | --- | | `/etc/ssl/private/cert.key`, `cert.pem` | `turnkey-ssl` postinst copies the shipped key here when none exists | yes, public key sha256 `139a0fd7...` | | `/usr/share/turnkey-ssl/cert.key`, `cert.pem` | the package payload, mode 0644 | yes, the same key | | `/etc/ssh/ssh_host_{rsa,ecdsa,ed25519}_key` | openssh-server postinst at build time | yes, ed25519 `SHA256:co8r09MoEC8FVPxSBL5ltte5n7uA3dvi8FeWS05+mUQ` | | `/etc/webmin/miniserv.pem` | webmin postinst at build time; unused, `miniserv.conf` points at `cert.pem` | yes | | `/etc/ssl/private/ssl-cert-snakeoil.key` | ssl-cert, in `lapp` and `postgresql` only | differs per layer, same in every image of that layer | So every image built from any published template starts with the same SSH host keys and the same TLS key in the place services read, until `10regen-sshkeys` and `15regen-sslcert` replace them. If either hook does not run, the machine serves a key anyone can download. The one public appliance is not affected: `forum` serves its own key on 443 and 12321 (`0ce6f58b...`) and has its own SSH host key. The fix, in two parts: 1. **Nothing ships.** `common/removelists-final/turnkey` removes all of the above as the last step of every layer build, and a dpkg `path-exclude` stops `turnkey-ssl` from putting its key back on upgrade. 2. **Generated before anything reads it.** A oneshot unit ordered before ssh, webmin, the web servers, postgresql and the init fence generates any key that is missing, on the machine. A service can then never start with a shared key or with none, whether the first boot hooks ran or not.

marcos-mendez commented

2026-09-29 01:36:35 -03:00

(Migrated from github.com)

The code is merged: Keel-Linux/inithooks#19 (keel-host-keys: every machine makes its own keys before a service reads one, and replaces any key that was published in an image), Keel-Linux/common#14 (the build removes them), Keel-Linux/buildtasks#14 (bt-layer refuses a layer that carries one). The security review's findings were fixed in the same PRs; the notable one was dropping the dpkg path-exclude, which would have broken turnkey-ssl's postinst.

Open until the published images are clean, which takes three attended steps in this order: publish inithooks 2.3.6+keel6; update /turnkey/fab-keel/common and /turnkey/buildtasks-keel on the build host together; rebuild the chain and publish. Rebuilding turnkey-ssl without its key stays with tracker#15.

The code is merged: Keel-Linux/inithooks#19 (`keel-host-keys`: every machine makes its own keys before a service reads one, and replaces any key that was published in an image), Keel-Linux/common#14 (the build removes them), Keel-Linux/buildtasks#14 (`bt-layer` refuses a layer that carries one). The security review's findings were fixed in the same PRs; the notable one was dropping the dpkg `path-exclude`, which would have broken turnkey-ssl's postinst. Open until the published images are clean, which takes three attended steps in this order: publish `inithooks 2.3.6+keel6`; update `/turnkey/fab-keel/common` and `/turnkey/buildtasks-keel` on the build host together; rebuild the chain and publish. Rebuilding turnkey-ssl without its key stays with tracker#15.