Decision note 0016: mirror layout, channels and rollback #10
No reviewers
Labels
No labels
accessibility
bug
documentation
duplicate
enhancement
good first issue
help wanted
invalid
question
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
Keel-Linux/handbook!10
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "docs/decision-0016-mirror-layout"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Closes #7.
Decision note 0016, and only 0016: 0014 and 0015 are being written
concurrently, so the number was taken deliberately rather than as "the next
free one".
What it records
Blobs content addressed and never overwritten or deleted; releases immutable
and identified by release and revision; channels the only mutable object,
signed and carrying a timestamp and an expiry; rollback as moving a pointer;
an expired channel an error at the client.
The two pieces of reasoning the issue asked for are written out rather than
left in a commit message:
overwrite each other, and with dated releases a rebuild for a security
update on the same day is exactly that case.
mirror freezes a client on a vulnerable release simply by not updating, and
without a timestamp inside the signature the client cannot tell that apart
from "nothing new".
Two things the note adds beyond the issue
Which key signs a channel, and what that key may say. The channel key is
the unattended one on the build host, today the staging key of decision 0011,
because re-signing before every expiry is daily and a design that needs the
maintainer at a terminal daily gets bypassed — by lengthening the expiry until
it means nothing. The note states the boundary plainly rather than implying a
guarantee we do not have: an online key closes the mirror, which is a
different machine holding no key, and does not close a compromised build host.
It also says explicitly that this extends 0011 rather than contradicting its
"no appliance may use the staging key" wording, because signing a pointer is a
strictly weaker claim than signing a release.
main. It stays the trunk and is always the latest; release branches arecut only when the next Debian opens, and the forks keep
masterso they staymergeable upstream.
What the review added to the note
The note's argument stood; four things it did not argue are now in it, each
one found by the review attacking the design rather than the prose.
online key is untrue: one signature pins an appliance to a known
vulnerable revision for ever and the rollback refusal never fires, which
is the exact failure mode the note offers as the reason a person cannot be
in the loop daily. Thirty days, enforced by the client.
revocation — did not work, because both implementations accepted a revoked
key. And a thief can delay the keyring that retires them by freezing
appliances, which the ceiling bounds. The keyring travels through the
archive, not through the channel it protects.
message that does not misdiagnose a fleet-wide NTP failure as a stale
mirror.
nothing ships it, so the pointer clients read expires seven days after the
last publication, not the last refresh. Answered as the open question it
is, with the interim rule, and with a note that this must not become an
argument for a longer ceiling.
Also: the "one file, not a file beside a detached signature" rationale moved
into the layout section where it belongs, with the consequence that makes it
matter — clear signing dash-escapes, so the bytes on the mirror are not the
bytes signed, which is why a reader must take the body from the verifier and
never parse the file. And the coverage figure is gone: a note is not the
place for a number it cannot qualify, and that number was compatible with the
verifier never running.
Also here: the traps
Both in
docs/traps.md, because each would have shipped a green check overcode that does not work.
that is now the most useful thing in it. It named a function that did not
implement the fix it described, and its stated rule — exit status plus
VALIDSIG— is not sufficient: a revoked or expired key gives exit 0 andVALIDSIG, withholding onlyGOODSIG. The entry now names the functionthat does implement it, carries the measured table, records that
VALIDSIG's last field is the primary so pinning a primary accepts arevoked subkey of it, and adds the relative-
--keyringcase that producesthe same
NO_PUBKEYas a wrong key. An entry recording a fix that is notthere costs more than a missing entry, so the entry says so.
work, in
apt/lib/mirror.sh, so that entry now draws the lesson it didnot: if a format's last field can contain the separator, counting fields
is the bug.
localdeclaration bash expands every word before assigning anyof them, so
local a="$1" b="$dir/$a"givesbthe valueahad outsidethe function. It hit twice in the publisher, and the second time the
generated script died at an unbound name under
set -ubefore printing therefusal it existed to print. The adversarial test found it; the happy path
passed.
Related
Reviewed alongside Keel-Linux/keel#34 and Keel-Linux/apt#14. Measured means run here on gpgv 2.4.7 / Debian 13.7 against those two branches.
On the question of whether the note argues the design or merely describes it: it argues. "Why the revision, and not the release alone" names the case a date cannot distinguish. "Why a signed pointer with an expiry is not optional" states the freeze attack and why a warning would not do. "Which key signs a channel" does the hardest job in the note — it reaches the uncomfortable conclusion, gives the operational reason (a daily passphrase gets bypassed by lengthening the expiry), states the boundary rather than implying a guarantee, and then reconciles itself with 0011 instead of quietly contradicting it. "What this does not decide" is three real open questions and not a formality. That is a decision note.
The findings below are about what it does not argue, and about
docs/traps.md.HIGH —
docs/traps.md: the gpgv entry records a fix that is not in the code it namesThe Fix paragraph says the output is read "only after the exit status and a
[GNUPG:] VALIDSIG <fpr>line naming an accepted key have both been accepted", and names two implementations:keel.layers.signature.verify_bytes, andchannel_verifiedinapt/lib/channel.sh.channel_verifiedisapt/lib/channel.sh:165-177. It runs— exit status, and nothing else. No
--status-fd, noVALIDSIG, no fingerprint. The trap file states as done a thing the code it cites does not do.apt/lib/release.sh:243-245is the function that actually implements the described fix, and it is not the one named.That is the failure mode this file exists to prevent, in this file. It also means a reader who trusts the entry will not look at
channel_verifiedagain.HIGH — the gpgv entry's fix is not sufficient anywhere, including where it is implemented
"Exit status and a VALIDSIG line" does not establish that a signature is good. Measured, gpgv 2.4.7:
GOODSIG,VALIDSIGREVKEYSIG,VALIDSIGREVKEYSIG,VALIDSIG— andVALIDSIG's last field is the primary fingerprintKEYEXPIRED,EXPKEYSIG,VALIDSIGstderr says
Good signature fromin all four. End to end on keel#34: a pointer signed by a key that is revoked in the very keyring passed to--channel-keyring, with--channel-signerpinned to its fingerprint, gives exit 0 and fetches the layer. Same for an expired key.So the entry records two properties of gpgv that shaped the design, and omits the two that break it. Given the standard this file sets — "each would have shipped a green check over code that does not work" — these belong in it more than the two that are there. The accurate rule is "require a
GOODSIGline", because gpgv emitsGOODSIGonly when the key is neither revoked nor expired and substitutesREVKEYSIG/EXPKEYSIGotherwise.This is not academic for this project:
apt/keys/keel-archive-keyring.ascalready carriessub D276B62C2BD16F4E … r, a signing subkey that has already been revoked once and rotated (keel-wordpress/COVERAGE.md:134-163). A third gpgv fact worth the same paragraph:--keyringwith a relative path is resolved against$GNUPGHOME, not the working directory, and a keyring it cannot find producesNO_PUBKEY— the same message as a wrong key, and the same message as the armored-keyring case the entry already records.MEDIUM — "and until when. That is all" understates the online key
Lines 99-102 are the narrow claim the design rests on, and line 116-118 concedes the weaker half: "whoever holds the online key can keep re-signing a stale pointer." Two things are missing from that concession.
First, they need not keep doing it. Nothing bounds
expires_atat either end. Measured on keel#34: a pointer signed now withexpires_at 2126-01-01T00:00:00Zis accepted, exit 0. On apt#14,KEEL_CHANNEL_TTL_DAYS=100000000producesexpires_at 275817-06-11and exits 0 silently. So one signature from the online key pins an appliance to a known-vulnerable, already-staged revision indefinitely, andCHANNEL_ROLLBACKnever fires because the revision does not go backwards.Second, that is precisely the failure mode the note itself names two paragraphs earlier — "by lengthening the expiry until it means nothing, which is worse than not having one" — offered as the reason a person must not be in the loop daily. The design identifies the lever, removes the human who would pull it, and then leaves the lever unguarded and unlogged for the unattended key. A client-side ceiling on
expires_at - signed_atcosts two comparisons and turns "forever" into "N days".While there: nothing refuses a
signed_atin the future either (measured:signed_at 2076-01-01accepted).MEDIUM — the note does not say how the channel key is retired
It puts a key online and argues well for why. It does not say what happens when that key is stolen, which for an unattended key on a build host is the event to plan for. The answer the reader will assume — revoke it, ship the revocation in the keyring — does not work, per the HIGH above. And the project has already done exactly this manoeuvre once, for the archive signing subkey.
A decision note that moves a key online owes a paragraph on retirement: revoke, rotate, and how appliances learn — including the awkward part, that the stolen key can delay the keyring that would retire it by freezing appliances on an old revision.
MEDIUM — nothing on clock skew
The expiry is checked client-side as
now >= expires_atagainst the appliance's own clock, and the note does not mention the clock once. Both directions matter and neither is obvious:nowand never mentions the local clock. The appliance is stalled, not bricked (--release/--revand the flat layout still work), but a fleet-wide NTP failure sends the operator to the mirror.One paragraph: the expiry is only as good as the clock, here is what each direction looks like, and here is why we accept it.
MEDIUM — "seven days, refreshed daily" describes an operational property the implementation does not have
Lines 120-128. On apt#14,
keel-channel refreshwrites$KEEL_RELEASE_ROOT/channelson the build host; the only thing that installs a pointer on the public host iskeel-publish-mirror, which requires a mandatory release date and an operator-opened tunnel. There is no timer, unit or script that ships a refreshed pointer. So the pointer clients read expires seven days after the last manual full release publication, and "six consecutive failures of the refresh are survivable and a seventh is not" measures a refresh no client ever sees. Either the note should say publication is manual, or apt#14 should grow a delivery path — I have raised it there as the finding that needs a design answer rather than a patch.LOW
keel/docs/layers.mdand not in the note. It is the right call and the reason given is right (and it is the same reason apt madeInRelease), but it is a decision about the mirror layout and this is the mirror-layout note. Worth a sentence beside "The layout", with the consequence: the bytes on the mirror are not the bytes signed, because clearsign dash-escapes and does not sign trailing whitespace, which is why the client must read gpgv's--outputand never the file. Both implementations get this right; neither the note nor the traps entry says why it matters.ln -fover an unvalidatedtarballfield and is one of two ways a published blob can currently be overwritten. Not the note's fault, but it is the note's sentence that makes "blobs are immutable" and "flat names are hard links" sound compatible, and today they are not.Miss=0,BrPart=0). It is also, on that branch, compatible with the CLI plumbing of--channel-signerbeing deleted on both paths with no test noticing, and with the whole gate passing green at 97% when gpgv is absent. A decision note is not the place for a coverage figure it cannot qualify; either drop it or say what it does and does not establish.docs/traps.mdalready carries "A path with a space in it breaks a whitespace comparison", whose fix is to key on a tab and to count paths with an internal space before trusting a count.apt/lib/mirror.sh:133isawk 'NF == 3 …'over MANIFEST lines, which silently drops any path containing a space from the digest check while the file is still fetched and installed. Pre-existing, but it is the same trap, and the file it is recorded in is the file this PR is editing.What is right and worth keeping as written
localtrap entry is accurate, correctly diagnosed, gives the two-statement fix, and says which test found it and that the happy path passed. I swept every shell file in every Keel repository and worktree with a quote-aware scanner validated against the known-bad shapes: zero remaining occurrences. The entry did its job.Warning. The note itself is sound and should land close to as written; the two HIGHs are both in
docs/traps.md. A traps file that names the wrong implementation, and whose stated fix does not hold, costs more than a missing entry — the next reader stops checking. Fix the gpgv entry to nameapt/lib/release.sh:243-245, to say thatchannel_verifiedstill needs the same treatment, and to record that exit 0 plusVALIDSIGis produced for revoked and expired keys, so the rule isGOODSIG. Then the MEDIUMs: a retirement paragraph, a TTL ceiling, a clock-skew paragraph, and either a caveat on "refreshed daily" or a delivery path in apt#14.25a6456and9f52bf7: the note and the gpgv trap now describe what apt#14 actually does after its fixes, the rollback section says how a revision is verified, the em dashes are gone, and the status line separates the maintainer's decision (layout, expiry) from the choice of channel key, which is key custody under BRIEF section 11 and still needs the maintainer.