The build reads the staging archive without verifying it #7

New issue

Closed

opened 2026-09-27 14:58:45 -03:00 by marcos-mendez · 1 comment

marcos-mendez commented

2026-09-27 14:58:45 -03:00

(Migrated from github.com)

Measured during a mariadb rebuild, 2026-09-27:

W: OpenPGP signature verification failed: file:/srv/keel-apt/repo
   trixie-staging InRelease: Missing key
   8CFD1A4841448B2227341CEB202CACBD0E97090A

The staging distribution was given its own key so a nightly could sign itself
and so appliance recipes could stop reading it through [trusted=yes], which
switches verification off. Half of that landed: staging is signed. The other
half did not: the build environment has no copy of the staging public key, so
apt warns and carries on unverified, which is where it was before, only now it
says so.

Closes when a layer build verifies the staging archive: the staging keyring is
installed where the build's apt reads keyrings, the source entry names it
through Signed-By, no [trusted=yes] remains in a recipe, and a build fails
rather than warns when the signature cannot be checked. The keyring must not
reach the finished image, the way the build time source does not:
common/removelists-final/turnkey is where that is arranged today.

Touches the recipes that copy the archive into their bootstrap, which is
keel-nodebb, keel-mariadb, keel-postgresql and keel-wordpress, all four
carrying the same block, and it is one more argument for the shared CI and
recipe pieces in tracker#4's neighbourhood.

Found while rebuilding mariadb for keel 0.3.5. Note for whoever takes it: a
rebuilt parent orphans its children, so mariadb and wordpress have to be
rebuilt and published together, which keel verify correctly refused to let
happen by halves.

Measured during a mariadb rebuild, 2026-09-27: W: OpenPGP signature verification failed: file:/srv/keel-apt/repo trixie-staging InRelease: Missing key 8CFD1A4841448B2227341CEB202CACBD0E97090A The staging distribution was given its own key so a nightly could sign itself and so appliance recipes could stop reading it through `[trusted=yes]`, which switches verification off. Half of that landed: staging is signed. The other half did not: the build environment has no copy of the staging public key, so apt warns and carries on unverified, which is where it was before, only now it says so. Closes when a layer build verifies the staging archive: the staging keyring is installed where the build's apt reads keyrings, the source entry names it through `Signed-By`, no `[trusted=yes]` remains in a recipe, and a build fails rather than warns when the signature cannot be checked. The keyring must not reach the finished image, the way the build time source does not: `common/removelists-final/turnkey` is where that is arranged today. Touches the recipes that copy the archive into their bootstrap, which is keel-nodebb, keel-mariadb, keel-postgresql and keel-wordpress, all four carrying the same block, and it is one more argument for the shared CI and recipe pieces in tracker#4's neighbourhood. Found while rebuilding mariadb for keel 0.3.5. Note for whoever takes it: a rebuilt parent orphans its children, so mariadb and wordpress have to be rebuilt and published together, which `keel verify` correctly refused to let happen by halves.

marcos-mendez commented

2026-09-27 17:58:55 -03:00

(Migrated from github.com)

Closed. The build verifies the staging archive, and the chain was rebuilt on it.

How it verifies. In all four recipes: the public half of the staging key is installed into the build tree as /etc/apt/keyrings/keel-staging-keyring.asc, from /srv/keel-apt/keys/keel-staging-keyring.asc, which bin/publish of keel-linux/apt now writes on every signed publication (apt#11); the source entry names it through signed-by; no recipe says trusted=yes any more; apt-get update runs with --error-on=any; and bin/keel-archive-check makes the same checks itself with gpgv, at the bootstrap and again on the tree about to be configured, where no apt-get update runs.

How it fails. Measured against the real archive in a scratch apt root:

tree	apt
`[trusted=yes]`, no keyring	`W: ... Missing key`, exit 0, packages installed
`signed-by`, no keyring	`E: The repository ... is not signed`, exit 100
`signed-by`, the wrong key	`E: The repository ... is not signed`, exit 100
`signed-by`, the staging key	`Get:1 ... InRelease`, exit 0

The keyring does not reach the image. The conf script removes it with the source entry and the archive copy; common/removelists-final/turnkey removes all three whatever a recipe does (Keel-Linux/common#4); conf.d/zzz-keel-archive refuses to enable the appliance's own source while either is present; and the wordpress boot test reads the assembled rootfs and fails if any of the three is there. Absent from both rootfs trees published tonight, and both delta tarballs carry the three paths as whiteouts.

The rebuild this was blocking. mariadb 0adca434 to c1d6fe68 and wordpress c9e33fa1 to c09dc984, rebuilt and published together with the verification in place; keel verify reports 6 checked, 6 ok on the staged tree and again against mirror.keellinux.org. The new mariadb carries keel 0.3.5 and conf.d/main raises the floor to it. appliance / build-and-boot on keel-mariadb#10 is green, the replication phase and its refusal included, and that pull request is merged.

Pull requests: Keel-Linux/common#4, apt#11, keel-mariadb#12 and #13, keel-postgresql#8 and #9, keel-nodebb#11 and #12, keel-wordpress#4.

The four way duplication this issue notes is now #8, and a keel-wordpress workflow that never runs on its own default branch is #9.

Closed. The build verifies the staging archive, and the chain was rebuilt on it. **How it verifies.** In all four recipes: the public half of the staging key is installed into the build tree as `/etc/apt/keyrings/keel-staging-keyring.asc`, from `/srv/keel-apt/keys/keel-staging-keyring.asc`, which `bin/publish` of keel-linux/apt now writes on every signed publication (apt#11); the source entry names it through `signed-by`; no recipe says `trusted=yes` any more; `apt-get update` runs with `--error-on=any`; and `bin/keel-archive-check` makes the same checks itself with `gpgv`, at the bootstrap and again on the tree about to be configured, where no `apt-get update` runs. **How it fails.** Measured against the real archive in a scratch apt root: | tree | apt | | --- | --- | | `[trusted=yes]`, no keyring | `W: ... Missing key`, exit 0, packages installed | | `signed-by`, no keyring | `E: The repository ... is not signed`, exit 100 | | `signed-by`, the wrong key | `E: The repository ... is not signed`, exit 100 | | `signed-by`, the staging key | `Get:1 ... InRelease`, exit 0 | **The keyring does not reach the image.** The conf script removes it with the source entry and the archive copy; `common/removelists-final/turnkey` removes all three whatever a recipe does (Keel-Linux/common#4); `conf.d/zzz-keel-archive` refuses to enable the appliance's own source while either is present; and the wordpress boot test reads the assembled rootfs and fails if any of the three is there. Absent from both rootfs trees published tonight, and both delta tarballs carry the three paths as whiteouts. **The rebuild this was blocking.** `mariadb` `0adca434` to **`c1d6fe68`** and `wordpress` `c9e33fa1` to **`c09dc984`**, rebuilt and published together with the verification in place; `keel verify` reports 6 checked, 6 ok on the staged tree and again against mirror.keellinux.org. The new `mariadb` carries keel 0.3.5 and `conf.d/main` raises the floor to it. `appliance / build-and-boot` on keel-mariadb#10 is green, the replication phase and its refusal included, and that pull request is merged. Pull requests: Keel-Linux/common#4, apt#11, keel-mariadb#12 and #13, keel-postgresql#8 and #9, keel-nodebb#11 and #12, keel-wordpress#4. The four way duplication this issue notes is now #8, and a `keel-wordpress` workflow that never runs on its own default branch is #9.