fix: images take packages from Debian and the Keel repository only #30

Merged

marcos-mendez merged 5 commits from fix/keel-apt-sources into 19.x

2026-10-02 14:53:33 -03:00

marcos-mendez commented

2026-10-02 10:07:51 -03:00

(Migrated from github.com)

Summary

The maintainer found that the first-boot security-updates step of Keel images still fetches from archive.turnkeylinux.org. This is where that comes from. The step8 Web image ships these files, all of them written by conf/bootstrap_apt and overlays/bootstrap_apt:

security.sources.sources: a TurnKey stanza (trixie-security, tkl-archive-keyring.gpg, enabled) next to Debian security. This is the file 95secupdates and cron-apt read.
sources.sources: TurnKey trixie (enabled) next to Debian trixie.
turnkey-testing.sources (disabled), with TurnKey's keys trusted.
/etc/apt/preferences: o=turnkeylinux at 999.

After this change (handbook decisions 0039 and 0043, tracker#23):

file	content
`debian.sources`	deb.debian.org trixie trixie-updates, `debian-archive-keyring.pgp`
`security.sources`	security.debian.org trixie-security, `debian-archive-keyring.pgp`
`debian-backports.sources`	unchanged, disabled
`keel.sources`	archive.keellinux.org trixie (stable track) enabled, trixie-testing disabled, `keel-archive-keyring.gpg`
`preferences.d/keel`	`Pin: release o=Keel Linux`, `Pin-Priority: 990`

plans/turnkey/base installs keel-archive-keyring instead of turnkey-keys. Fingerprint AD0964BE3F09DED469A3B6B2148E951314703180.
Why 990. It is above Debian's 500, so a Keel package is the candidate whatever version Debian has. It is below 1000, so apt never installs it over a newer installed version. At 1001 it does (tracker#23). tests/apt-sources.bats asserts both against apt with local archives, and fails at 1001 and at 500 (checked by mutation).
conf/turnkey.d/keel-apt stops the build when:
- keel.sources has no enabled https://archive.keellinux.org trixie stanza;
- keel-archive-keyring is not installed, or is older than 0.1.1;
- the pin on o=Keel Linux is missing, or anything but Pin-Priority: 990, in /etc/apt/preferences or any file of preferences.d. A recipe overlay's preferences.d/keel at 1001 is caught. The build's own o=Keel Linux Pool pin and the recipes' build-only l=Keel Linux staging pin are not matched; the final removelist takes both out.
- anything still names TurnKey: /etc/apt/sources.list, sources.list.d, preferences, preferences.d, an unowned tkl-* keyring, or a key in trusted.gpg or trusted.gpg.d whose user id is TurnKey's (read with gpg, so a renamed file is caught).
A turnkey-keys that a parent layer installed is purged with dpkg -P. Only the source files TurnKey's bootstrap wrote are removed by name.
bootstrap_apt removes the three names it used to write. Measured on the step8 image: left next to debian.sources, sources.sources makes apt refuse every source ("Conflicting values set for option Signed-By").
Goes with Keel-Linux/inithooks#34 (95secupdates reads security.sources). Both must be in the same image build. An old inithooks with this common would upgrade nothing on first boot, because apt treats a missing sourcelist as empty.

The pool requirement (met, 2026-10-02)

Requirement. The captured pool that the next Core build is pointed at (keel-pool current) must hold keel-archive-keyring, Architecture all, at 0.1.1 or newer: conf/turnkey.d/keel-apt refuses an older one, since 0.1.0 shipped the revoked signing subkey. The package installs /usr/share/keyrings/keel-archive-keyring.gpg, primary key fingerprint AD0964BE3F09DED469A3B6B2148E951314703180. A pool without it fails the Core build at the plan (keel-archive-keyring not found), and every layer above Core inherits that.

What archive.keellinux.org serves now (read on 2026-10-02, after the release work published it):

suite	package	version	.deb SHA256
`trixie-testing`	keel-archive-keyring	0.2.0	`927d49e88d2e1753d086bf4eb24abd7caedd6cc2ef0db470f5c697f550165399`
`trixie-testing`	turnkey-ssl	3.1.1+keel1	`1fff5fd6503ed06b3beee86bc2cf9f64fe639328d87247851ae9ad4a3bd31995`
`trixie-testing`	turnkey-netinfo	1.1.1+keel2	`d435cca4d8c061b02181f007cfc00a2750b8d641c390178f6a046359d3fa1f2b`
`trixie-testing`	turnkey-conffile	1.1.2+keel1	`10b669a879000f0ba4af9cfd31389566324911be8557556a5fd4e7177740b0c5`
`trixie`	keel-archive-keyring	0.1.1	`15ea1c3964da36d39808f64f42f6280c24ce3faedded9eb2f06489804545ff93`

keel-archive-keyring.asc answers 200 at the archive root. The next Core build therefore captures keel-archive-keyring 0.2.0 from trixie-testing, and keel-apt accepts it (0.2.0 is above the 0.1.1 floor). The image's own keel.sources is unchanged by this: it still follows trixie, the stable track, with trixie-testing present but disabled. Which track a build reads is the release work's (decision 0039).

Earlier measurement on the build host, 2026-10-02, kept for the record:

The current pool, /srv/keel-pool/dists/2026-09-27, has 411 members captured from core. They include turnkey-keys 0.1 and TurnKey's own inithooks 2.3.6 and confconsole 2.2.3. No Keel package is in it, keel-archive-keyring included.
keel-pool capture fetches with the build host's own apt-get download --print-uris. The host's apt sources are deb.debian.org, security.debian.org and http://archive.turnkeylinux.org/debian. The Keel archive is not among them, so a capture cannot fetch keel-archive-keyring and would record it as a gap.
At that time only keel-archive-keyring 0.1.1 was published, in trixie and trixie-staging, and the build host's apt sources did not include the Keel archive, so a capture could not fetch it. Both are resolved by the publication above and the release work's build host setup.

The seven recipes, which merge first

Seven recipes shipped their own keel.sources (disabled) and preferences.d/keel (1001) at common's paths, and asserted Enabled: no. Those builds fail on this PR until these merge:

Each drops both overlay files. The 1001 pin had also been what made the build-time staging archive beat TurnKey's 999 pin during the recipe's upgrade, so each recipe now pins l=Keel Linux staging at 1001 for the build only (/etc/apt/preferences.d/keel-staging) and removes it with the staging source. This PR's final removelist removes it as well. keel-core and keel-web have neither file.

Merge order: the seven recipes, then the pool, then this PR together with inithooks#34.

TurnKey-built packages: no updates any more (review item 5)

archive.keellinux.org trixie has 6 packages. The step8 Web image has 36 packages whose name and version match TurnKey's trixie or trixie-security index exactly (checked against both indexes today). Without turnkey-keys, which this PR purges, that is 35:

webmin and 20 modules, all 2.660.turnkey0: authentic-theme, custom, fail2ban, fdisk, filemin, firewall, firewall6, logviewer, lvm, mount, net, passwd, postfix, raid, software, sshd, systemd, updown, useradmin, xterm. Also webmin-tklbam 1.2.1.
tklbam 1.5.3+2+g31134d3, tklbam-squid and tklbam-squid-common 2.7.STABLE9-2.3.0-2turnkey+0, turnkey-pypy2 7.3.23+turnkey1.
turnkey-conffile 1.1.2, turnkey-netinfo 1.1.1, turnkey-ssl 3.1.1, turnkey-sysinfo 1.1.0, turnkey-version 1.1.3, hubdns 1.4.0, py3curl-wrapper 2.1.2, tkl-dhcpcd-ifupdown-glue 0.1.3, tkl-installer 0.1.2.

Update, 2026-10-02: the first three are now Keel forks in trixie-testing (turnkey-ssl 3.1.1+keel1, turnkey-netinfo 1.1.1+keel2, turnkey-conffile 1.1.2+keel1; see the table above), so 32 remain TurnKey's builds. At 990 a +keelN fork is the candidate over TurnKey's version of the same package, which is exactly the case the pin exists for.

inithooks and confconsole are already Keel builds (+keelN).

Proposal (follow-up in Keel-Linux/apt): bin/keel-mirror-turnkey. It would publish those exact versions into Keel's trixie, so updates reach them through the Keel archive:

Fetch them from archive.turnkeylinux.org trixie and trixie-security. Verify each .deb's SHA256 against TurnKey's signed Release, checked against TurnKey's key on the build host only and never on an appliance, plus its control fields. lib/pool.sh (pool_uris, pool_check) already does this per package.
Include them into trixie with reprepro, signed with the Keel key. Each publication is one commit listing the packages.
Run it on a schedule, and add a package only when its version moves.

In the long run each one is rebuilt from source as ours (tracker#15). turnkey-ssl is the first candidate, because it ships a private key (keel-core#8).

Other notes

trixie-testing is published now (it answered 404 when this PR was opened). The image's stanza for it ships disabled: a simple installation follows stable (decision 0039).
Rebased onto 19.x after common#31 (root locked at export, /etc/keel/build-date). The only conflict was the shared unreleased changelog entry; both bullet groups are kept, #31's first. Goes with inithooks#34, rebased likewise after inithooks#35, whose entry is now 2.3.6+keel18 above #35's keel17.
The migration for existing machines is Keel-Linux/keel-transition#4: pin 990, Debian's sources in place of TurnKey's, the 999 pin set aside, turnkey-keys purged, all reversible.

Verification

The step8 Web image was patched in build order in an LXC container: bootstrap_apt, then keel-archive-keyring in place of turnkey-keys, then the overlay, then the conf scripts, then the new inithooks.

apt update contacted only deb.debian.org, security.debian.org and archive.keellinux.org. The Keel archive shows at 990 in apt-cache policy.
apt-get -s upgrade and apt-get -s dist-upgrade showed 0 downgrades. The image's inithooks 2.3.6+keel16, confconsole +keel11 and keel 0.15.1 are kept over the archive's keel5, keel2 and 0.3.5. With the pin set to 1001, dist-upgrade downgrades those 3 packages.
turnkey-install-security-updates: with libpng16-16t64 rolled back to the trixie version, the hook fetched 1.6.48-1+deb13u6 from security.debian.org.

Test plan

bats tests/apt-sources.bats tests/apt-identity.bats: 45 pass (the 990 pin check and the removelist RED first). The review fixes were written RED first, and replacing the purge with rm fails both purge tests.
shellcheck on bootstrap_apt, keel-apt, cronapt and the tests
kcov: conf/turnkey.d/keel-apt at 100 percent (44 of 44 lines)
CI green
the next Core/Web build reads the pool, and its image passes conf/turnkey.d/keel-apt

## Summary The maintainer found that the first-boot security-updates step of Keel images still fetches from archive.turnkeylinux.org. This is where that comes from. The step8 Web image ships these files, all of them written by `conf/bootstrap_apt` and `overlays/bootstrap_apt`: - `security.sources.sources`: a TurnKey stanza (trixie-security, `tkl-archive-keyring.gpg`, enabled) next to Debian security. This is the file 95secupdates and cron-apt read. - `sources.sources`: TurnKey trixie (enabled) next to Debian trixie. - `turnkey-testing.sources` (disabled), with TurnKey's keys trusted. - `/etc/apt/preferences`: `o=turnkeylinux` at 999. After this change (handbook decisions 0039 and 0043, tracker#23): | file | content | |---|---| | `debian.sources` | deb.debian.org trixie trixie-updates, `debian-archive-keyring.pgp` | | `security.sources` | security.debian.org trixie-security, `debian-archive-keyring.pgp` | | `debian-backports.sources` | unchanged, disabled | | `keel.sources` | archive.keellinux.org trixie (stable track) enabled, trixie-testing disabled, `keel-archive-keyring.gpg` | | `preferences.d/keel` | `Pin: release o=Keel Linux`, `Pin-Priority: 990` | - `plans/turnkey/base` installs `keel-archive-keyring` instead of `turnkey-keys`. Fingerprint AD0964BE3F09DED469A3B6B2148E951314703180. - **Why 990.** It is above Debian's 500, so a Keel package is the candidate whatever version Debian has. It is below 1000, so apt never installs it over a newer installed version. At 1001 it does (tracker#23). `tests/apt-sources.bats` asserts both against apt with local archives, and fails at 1001 and at 500 (checked by mutation). - `conf/turnkey.d/keel-apt` stops the build when: - keel.sources has no enabled `https://archive.keellinux.org trixie` stanza; - keel-archive-keyring is not installed, or is older than 0.1.1; - the pin on `o=Keel Linux` is missing, or anything but `Pin-Priority: 990`, in `/etc/apt/preferences` or any file of `preferences.d`. A recipe overlay's `preferences.d/keel` at 1001 is caught. The build's own `o=Keel Linux Pool` pin and the recipes' build-only `l=Keel Linux staging` pin are not matched; the final removelist takes both out. - anything still names TurnKey: `/etc/apt/sources.list`, `sources.list.d`, `preferences`, `preferences.d`, an unowned `tkl-*` keyring, or a key in `trusted.gpg` or `trusted.gpg.d` whose user id is TurnKey's (read with gpg, so a renamed file is caught). A `turnkey-keys` that a parent layer installed is purged with `dpkg -P`. Only the source files TurnKey's bootstrap wrote are removed by name. - `bootstrap_apt` removes the three names it used to write. Measured on the step8 image: left next to debian.sources, `sources.sources` makes apt refuse every source ("Conflicting values set for option Signed-By"). - Goes with Keel-Linux/inithooks#34 (95secupdates reads `security.sources`). Both must be in the same image build. An old inithooks with this common would upgrade nothing on first boot, because apt treats a missing sourcelist as empty. ## The pool requirement (met, 2026-10-02) **Requirement.** The captured pool that the next Core build is pointed at (`keel-pool current`) must hold **`keel-archive-keyring`, Architecture `all`, at 0.1.1 or newer**: `conf/turnkey.d/keel-apt` refuses an older one, since 0.1.0 shipped the revoked signing subkey. The package installs `/usr/share/keyrings/keel-archive-keyring.gpg`, primary key fingerprint AD0964BE3F09DED469A3B6B2148E951314703180. A pool without it fails the Core build at the plan (`keel-archive-keyring` not found), and every layer above Core inherits that. **What archive.keellinux.org serves now** (read on 2026-10-02, after the release work published it): | suite | package | version | .deb SHA256 | |---|---|---|---| | `trixie-testing` | keel-archive-keyring | 0.2.0 | `927d49e88d2e1753d086bf4eb24abd7caedd6cc2ef0db470f5c697f550165399` | | `trixie-testing` | turnkey-ssl | 3.1.1+keel1 | `1fff5fd6503ed06b3beee86bc2cf9f64fe639328d87247851ae9ad4a3bd31995` | | `trixie-testing` | turnkey-netinfo | 1.1.1+keel2 | `d435cca4d8c061b02181f007cfc00a2750b8d641c390178f6a046359d3fa1f2b` | | `trixie-testing` | turnkey-conffile | 1.1.2+keel1 | `10b669a879000f0ba4af9cfd31389566324911be8557556a5fd4e7177740b0c5` | | `trixie` | keel-archive-keyring | 0.1.1 | `15ea1c3964da36d39808f64f42f6280c24ce3faedded9eb2f06489804545ff93` | `keel-archive-keyring.asc` answers 200 at the archive root. The next Core build therefore captures keel-archive-keyring 0.2.0 from `trixie-testing`, and `keel-apt` accepts it (0.2.0 is above the 0.1.1 floor). The image's own `keel.sources` is unchanged by this: it still follows `trixie`, the stable track, with `trixie-testing` present but disabled. Which track a build reads is the release work's (decision 0039). Earlier measurement on the build host, 2026-10-02, kept for the record: - The current pool, `/srv/keel-pool/dists/2026-09-27`, has 411 members captured from core. They include `turnkey-keys 0.1` and TurnKey's own `inithooks 2.3.6` and `confconsole 2.2.3`. No Keel package is in it, `keel-archive-keyring` included. - `keel-pool capture` fetches with the build host's own `apt-get download --print-uris`. The host's apt sources are deb.debian.org, security.debian.org and `http://archive.turnkeylinux.org/debian`. The Keel archive is not among them, so a capture cannot fetch keel-archive-keyring and would record it as a gap. - At that time only keel-archive-keyring **0.1.1** was published, in `trixie` and `trixie-staging`, and the build host's apt sources did not include the Keel archive, so a capture could not fetch it. Both are resolved by the publication above and the release work's build host setup. ## The seven recipes, which merge first Seven recipes shipped their own `keel.sources` (disabled) and `preferences.d/keel` (1001) at common's paths, and asserted `Enabled: no`. Those builds fail on this PR until these merge: - Keel-Linux/keel-lamp#5 - Keel-Linux/keel-lapp#5 - Keel-Linux/keel-wordpress#16 - Keel-Linux/keel-mariadb#21 - Keel-Linux/keel-postgresql#13 - Keel-Linux/keel-redis#11 - Keel-Linux/keel-nodebb#18 Each drops both overlay files. The 1001 pin had also been what made the build-time staging archive beat TurnKey's 999 pin during the recipe's upgrade, so each recipe now pins `l=Keel Linux staging` at 1001 for the build only (`/etc/apt/preferences.d/keel-staging`) and removes it with the staging source. This PR's final removelist removes it as well. keel-core and keel-web have neither file. Merge order: the seven recipes, then the pool, then this PR together with inithooks#34. ## TurnKey-built packages: no updates any more (review item 5) archive.keellinux.org trixie has 6 packages. The step8 Web image has **36 packages** whose name and version match TurnKey's trixie or trixie-security index exactly (checked against both indexes today). Without `turnkey-keys`, which this PR purges, that is 35: - `webmin` and 20 modules, all `2.660.turnkey0`: authentic-theme, custom, fail2ban, fdisk, filemin, firewall, firewall6, logviewer, lvm, mount, net, passwd, postfix, raid, software, sshd, systemd, updown, useradmin, xterm. Also `webmin-tklbam 1.2.1`. - `tklbam 1.5.3+2+g31134d3`, `tklbam-squid` and `tklbam-squid-common 2.7.STABLE9-2.3.0-2turnkey+0`, `turnkey-pypy2 7.3.23+turnkey1`. - `turnkey-conffile 1.1.2`, `turnkey-netinfo 1.1.1`, `turnkey-ssl 3.1.1`, `turnkey-sysinfo 1.1.0`, `turnkey-version 1.1.3`, `hubdns 1.4.0`, `py3curl-wrapper 2.1.2`, `tkl-dhcpcd-ifupdown-glue 0.1.3`, `tkl-installer 0.1.2`. Update, 2026-10-02: the first three are now Keel forks in `trixie-testing` (turnkey-ssl 3.1.1+keel1, turnkey-netinfo 1.1.1+keel2, turnkey-conffile 1.1.2+keel1; see the table above), so 32 remain TurnKey's builds. At 990 a `+keelN` fork is the candidate over TurnKey's version of the same package, which is exactly the case the pin exists for. inithooks and confconsole are already Keel builds (+keelN). **Proposal (follow-up in Keel-Linux/apt): `bin/keel-mirror-turnkey`.** It would publish those exact versions into Keel's `trixie`, so updates reach them through the Keel archive: 1. Fetch them from archive.turnkeylinux.org trixie and trixie-security. Verify each .deb's SHA256 against TurnKey's signed `Release`, checked against TurnKey's key on the build host only and never on an appliance, plus its control fields. `lib/pool.sh` (`pool_uris`, `pool_check`) already does this per package. 2. Include them into `trixie` with reprepro, signed with the Keel key. Each publication is one commit listing the packages. 3. Run it on a schedule, and add a package only when its version moves. In the long run each one is rebuilt from source as ours (tracker#15). `turnkey-ssl` is the first candidate, because it ships a private key (keel-core#8). ## Other notes - `trixie-testing` is published now (it answered 404 when this PR was opened). The image's stanza for it ships disabled: a simple installation follows stable (decision 0039). - Rebased onto 19.x after common#31 (root locked at export, `/etc/keel/build-date`). The only conflict was the shared unreleased changelog entry; both bullet groups are kept, #31's first. Goes with inithooks#34, rebased likewise after inithooks#35, whose entry is now 2.3.6+keel18 above #35's keel17. - The migration for existing machines is Keel-Linux/keel-transition#4: pin 990, Debian's sources in place of TurnKey's, the 999 pin set aside, turnkey-keys purged, all reversible. ## Verification The step8 Web image was patched in build order in an LXC container: bootstrap_apt, then keel-archive-keyring in place of turnkey-keys, then the overlay, then the conf scripts, then the new inithooks. - `apt update` contacted only deb.debian.org, security.debian.org and archive.keellinux.org. The Keel archive shows at 990 in `apt-cache policy`. - `apt-get -s upgrade` and `apt-get -s dist-upgrade` showed 0 downgrades. The image's inithooks 2.3.6+keel16, confconsole +keel11 and keel 0.15.1 are kept over the archive's keel5, keel2 and 0.3.5. With the pin set to 1001, dist-upgrade downgrades those 3 packages. - `turnkey-install-security-updates`: with libpng16-16t64 rolled back to the trixie version, the hook fetched 1.6.48-1+deb13u6 from security.debian.org. ## Test plan - [x] `bats tests/apt-sources.bats tests/apt-identity.bats`: 45 pass (the 990 pin check and the removelist RED first). The review fixes were written RED first, and replacing the purge with `rm` fails both purge tests. - [x] shellcheck on bootstrap_apt, keel-apt, cronapt and the tests - [x] kcov: conf/turnkey.d/keel-apt at 100 percent (44 of 44 lines) - [ ] CI green - [ ] the next Core/Web build reads the pool, and its image passes `conf/turnkey.d/keel-apt`