fix: every member learns the root CA's holder from a notice the root signed (keel#105) #112

Merged

marcos-mendez merged 2 commits from fix/root-holder-for-every-member into main

2026-10-08 22:44:16 -03:00

marcos-mendez commented

2026-10-08 21:52:25 -03:00

(Migrated from github.com)

Fixes #105 (bug B1 of the real-node MariaDB pair test, Keel-Linux/handbook#60).

Stacked on #111 (both bump debian/changelog): this branch is rebased on it and its base is #111's branch. GitHub moves the base to main when #111 merges.

Problem

In a mesh whose root CA is held by a cloud advanced node (web-1) that is not a pair member, a cloud simple pair cannot get its database certificate. Only etcd members write /var/lib/keel/etcd/holder, so each pair member asks the other one, which refuses: this node does not hold the mesh's root CA. The tester wrote web-1's address into that file by hand.

Which source is trusted

Decision 0051: "The holder's address is not in the bundle ... it is a short notice signed by that root's key alone". The root certificate is the anchor (the trust bundle of epoch 1).

The notice (mesh id, holder's overlay address, time) is signed by the root's key. Any roster may carry it, the holder's own or relayed, because only the root's key makes one. A member takes it only when it is for its mesh, newer than the one it keeps, and verifies with the root certificate it holds. A member that does not hold the root's key cannot send certificate requests to another address.
A member that holds no root certificate (no etcd member) takes one only from a roster it fetched from a trust root's own overlay address, as docs/mesh.md says a trust root's signing key is learned; never from an announcement. It must be a self-signed CA named for the mesh, and it is never replaced (a new root is 0051's bundle, by a majority).
A notice dated more than 5 minutes ahead of the local clock is refused: a notice dated far ahead would make every later one "older" and pin the address. A time and not a sequence number, because 0051 names the time and a counter would have to move with the root's key when the holder moves.
The notice has a region field (0051); absent means the first region. A notice of another region is not taken until the bundle exists.
Open points (old notice replayed to a new node, a trust root that gives a fake root, the root in the join answer): #114.

Change

keel/mesh/rootholder.py (new): the notice, made once on the holder and kept; learn verifies and keeps it, writes holder, holder.notice and root-anchor.crt; asked fetches the peers' rosters.
keel/mesh/members.py: the roster carries root and holder; a value that cannot be read is left out. Older members ignore the new fields.
keel/mesh/sync.py: the roster offers them; keel mesh sync and announcements take them.
keel/system/dbtls.py: a node that knows no holder fetches its peers' rosters before it asks, and asks the other pair member only when no roster names the holder. The leaf is checked against the learned root.
docs/mesh.md, docs/replication.md.
debian/changelog, pyproject.toml, keel/__init__.py: 0.23.5.

Tests

tests/test_mesh_rootholder.py: a pair of two cloud simple nodes, the root held by a third node, gets both database leaves with no file written by hand (through the roster wire format); keel mesh sync learns it, also relayed by the other pair member; a forged root with the mesh root's name and a notice signed by it are refused; a root from a member that is no trust root, or from an announcement, is not taken; a leaf under another root is refused; an older notice, one of another mesh, one more than 5 minutes ahead, and one of another region are not taken; the region is signed.

Test plan

tests / coverage
mesh / trixie (the roster on the real channel)
mariadb, vip, vip-upgrade, etcd, etcd-auth / trixie
package / changelog

CI record

First run: vip / trixie failed at g_promote_b (etcdctl get: context deadline exceeded, as in main's run 37833124848), tests / coverage failed in tests/test_cli_channel.py (a gpg call, not this change), and mariadb / trixie failed in #111's case (fixed there). No job was rerun: the rebase on #111 ran all checks again, all green.

Fixes #105 (bug B1 of the real-node MariaDB pair test, Keel-Linux/handbook#60). Stacked on #111 (both bump `debian/changelog`): this branch is rebased on it and its base is #111's branch. GitHub moves the base to `main` when #111 merges. ## Problem In a mesh whose root CA is held by a cloud advanced node (web-1) that is not a pair member, a cloud simple pair cannot get its database certificate. Only etcd members write `/var/lib/keel/etcd/holder`, so each pair member asks the other one, which refuses: `this node does not hold the mesh's root CA`. The tester wrote web-1's address into that file by hand. ## Which source is trusted Decision 0051: "The holder's address is not in the bundle ... it is a short notice signed by that root's key alone". The root certificate is the anchor (the trust bundle of epoch 1). - The notice (mesh id, holder's overlay address, time) is signed by the root's key. Any roster may carry it, the holder's own or relayed, because only the root's key makes one. A member takes it only when it is for its mesh, newer than the one it keeps, and verifies with the root certificate it holds. A member that does not hold the root's key cannot send certificate requests to another address. - A member that holds no root certificate (no etcd member) takes one only from a roster it fetched from a trust root's own overlay address, as docs/mesh.md says a trust root's signing key is learned; never from an announcement. It must be a self-signed CA named for the mesh, and it is never replaced (a new root is 0051's bundle, by a majority). - A notice dated more than 5 minutes ahead of the local clock is refused: a notice dated far ahead would make every later one "older" and pin the address. A time and not a sequence number, because 0051 names the time and a counter would have to move with the root's key when the holder moves. - The notice has a `region` field (0051); absent means the first region. A notice of another region is not taken until the bundle exists. - Open points (old notice replayed to a new node, a trust root that gives a fake root, the root in the join answer): #114. ## Change - `keel/mesh/rootholder.py` (new): the notice, made once on the holder and kept; `learn` verifies and keeps it, writes `holder`, `holder.notice` and `root-anchor.crt`; `asked` fetches the peers' rosters. - `keel/mesh/members.py`: the roster carries `root` and `holder`; a value that cannot be read is left out. Older members ignore the new fields. - `keel/mesh/sync.py`: the roster offers them; `keel mesh sync` and announcements take them. - `keel/system/dbtls.py`: a node that knows no holder fetches its peers' rosters before it asks, and asks the other pair member only when no roster names the holder. The leaf is checked against the learned root. - `docs/mesh.md`, `docs/replication.md`. - `debian/changelog`, `pyproject.toml`, `keel/__init__.py`: 0.23.5. ## Tests `tests/test_mesh_rootholder.py`: a pair of two cloud simple nodes, the root held by a third node, gets both database leaves with no file written by hand (through the roster wire format); `keel mesh sync` learns it, also relayed by the other pair member; a forged root with the mesh root's name and a notice signed by it are refused; a root from a member that is no trust root, or from an announcement, is not taken; a leaf under another root is refused; an older notice, one of another mesh, one more than 5 minutes ahead, and one of another region are not taken; the region is signed. ## Test plan - [ ] tests / coverage - [ ] mesh / trixie (the roster on the real channel) - [ ] mariadb, vip, vip-upgrade, etcd, etcd-auth / trixie - [ ] package / changelog ## CI record First run: `vip / trixie` failed at `g_promote_b` (`etcdctl get: context deadline exceeded`, as in main's run 37833124848), `tests / coverage` failed in `tests/test_cli_channel.py` (a gpg call, not this change), and `mariadb / trixie` failed in #111's case (fixed there). No job was rerun: the rebase on #111 ran all checks again, all green.