feat: etcd's certificates are the root's, and etcd's auth is on (#83) #90

Merged

marcos-mendez merged 1 commit from feat/etcd-root-issued-rbac into main

2026-10-08 06:39:35 -03:00

marcos-mendez commented

2026-10-08 04:14:47 -03:00

(Migrated from github.com)

Implements #83, which the maintainer approved on 2026-10-07. Packaging: Keel-Linux/common#48. Handbook amendment: Keel-Linux/handbook#56.

What changes

The root signs every etcd certificate itself. The inviter relays the request at a join (issue), and the member sends its own at a renewal. The CN is the member's mesh identity (keel-<address>) and the SANs are its /128 and ::1. The root sets both, whatever the request asks. The member checks key, name, address, CA:FALSE and the chain to the root. No member holds a CA key. Renewal (with 10 of 30 days left) and new memberships wait, with an alert, while the holder is unreachable. Issuance goes through the holder named in the member's state, which leaves room for 0051's other roots.
Who gets a certificate (keel.mesh.etcdproof, after the security review). The root signs only a request shown to be the member's own. The CSR's own signature is proof of possession of the CSR key (checked before signing); the member's Ed25519 signature over the CSR with its mesh signing key binds that key to the member. A member renews its own address itself (sender = that address's key, proof verified with the trust store's signing key for it). A node being admitted is relayed by its inviter with the admission evidence it signed (names the node, address and signing key; signed by the sender's trusted key); the proof must verify with the evidence's signing key. Evidence naming the inviter's own or any other member's signing key is refused; where the trust store knows the node, the evidence must name the same key; a certificate issued under a signing key the trust store later disowns is revoked at tend (etcdca.impostors). The holder records the signing key and the relaying member with each certificate, and the relay may revoke it, so abandoned() gives a failed join's certificate and learner back. tests/test_mesh_etcd_root.py now refuses B for C's address in every form; tests/etcd_auth_netns.py does the same on the real cluster.

etcd auth. Users are the CNs.

Role	Permission	Granted to
`keel-member`	read `/keel/<mesh>/`, write `/keel/<mesh>/etcd/` (the mesh's own keys, e.g. #87's restart lock)	every member
`keel-vip-<vip>`	read-write `/keel/<mesh>/vip/<vip>/`	the pair's two members only

The holder manages users and roles with an admin certificate (CN root) that only it holds (keel.mesh.etcdauth). Whether auth is on is read from etcd (auth status, cached 5 s), never from a file; every step after auth enable in reissue is idempotent, so a crash between them resumes on rerun. keel vip pair and tend send a pair's record to the holder, which checks it like any node does. Membership changes need root once auth is on, so they now run on the holder: learner promote and remove in tend, and member removal inside revoke.

etcdctl instead of the JSON gateway (keel.mesh.etcdclient). The gateway cannot carry a client's CN (etcd's docs, and tested on trixie's 3.5.16). Details:
- argv lists, never a shell;
- certificates by path; the VIP controller passes its memfds as /proc/self/fd/N via pass_fds;
- no secrets in argv;
- env is only PATH and GOMAXPROCS (fits the controller's TasksMax);
- -w json, etcd's own timeouts, plus a process kill at +0.5 s;
- a failed or unparseable call raises: it is never a renewal. Only etcd's own "requested lease not found" counts as a gone lease.
The gateway is now off (ETCD_ENABLE_GRPC_GATEWAY=false). keel-vip.service's helper rewrites the controller's memfds when the certificate changes (renewal, reissue). Before this, a controller running past 30 days kept a stale certificate.
CRL: still signed with the root key on the holder and distributed as before. Members sign nothing. The 0051 note is in handbook#56.

Migration of the live layout: `keel mesh etcd reissue` (on the root holder)

Checks, changing nothing: every member answers a probe and runs this keel (PKI 2), and every voter is healthy. --dry-run stops here and prints the plan.
Issues the holder's admin certificate and records its own intermediate.
One member at a time, the holder last: enroll, then the root signs, then reissue. The member drops its intermediate. etcd reads the new certificate at its next handshake, so nothing restarts. The next step waits until that member serves the new certificate and every voter is healthy (3 min max, then it stops with the reason; rerun continues).
Revokes every intermediate in the root's CRL and pushes the CRL to every member (crl).
Converges users and roles (including pairs taken from existing VIP claims), runs auth enable as root, then reads the mesh as itself to check.

Rollback: keel mesh etcd reissue --rollback disables auth. Certificates stay root-issued: the intermediates are revoked, which is the point. A run that stopped part way needs no rollback: both certificate kinds work together until step 4, and a rerun finishes the job.

Upgrade order: upgrade keel and keel-overlay-etcd on every node first, one node at a time (applying the new config restarts etcd once for the gateway change). keel Breaks: keel-overlay-etcd (<< 0.4.0~) so etcdctl comes with keel. Then run reissue once.

Known limits

etcd 3.5 runs no permission check on LeaseKeepAlive. A member outside the pair can keep a VIP lease alive by its ID, which can delay a failover but can never create a second holder (the VIP's checks are unchanged; docs/vip.md).
Rebased on #87, now merged: its multi-endpoint controller and etcd's gate go through etcdctl too. The gate's lock is under /keel/<mesh>/etcd/, which every member may write. This is keel 0.22.0.
common#46 (merged) took keel-overlay-etcd 0.3.0; common#48 is 0.4.0, and the Breaks here names it.

Tests

Unit: recording etcdctl, fake etcd with auth, reissue on a cluster of the old layout including every refusal path, receiver kinds, memfd refresh. Coverage stays above the 95% gate.
etcd / trixie and vip / trixie now run on etcdctl.
New etcd-auth / trixie (tests/test_etcd_auth_netns.py): 250 ms ±25 ms with 2% loss, measured first. Three members start in the live layout while the VIP is held at A, and reissue runs. It asserts:
- one leader and one term, and reads keep working on every member (quorum);
- the VIP holder stays A at every 200 ms sample, the ping gap is under 1.5 s, and the epoch is unchanged;
- every member is root-issued, every intermediate is revoked everywhere, and auth is on;
- C (not in the pair) gets permission denied on put, delete, revoke and txn of the VIP keys, while B passes;
- no CA key exists on any member, and a CN=root certificate signed with C's own key does not reach etcd;
- C's renewal through the root works, waits while A is cut off, and works again afterwards;
- rollback turns auth off.
It also measures etcdctl's overhead against the controller's 2 s call timeout.

Implements #83, which the maintainer approved on 2026-10-07. Packaging: Keel-Linux/common#48. Handbook amendment: Keel-Linux/handbook#56. ## What changes - **The root signs every etcd certificate itself.** The inviter relays the request at a join (`issue`), and the member sends its own at a renewal. The CN is the member's mesh identity (`keel-<address>`) and the SANs are its /128 and `::1`. The root sets both, whatever the request asks. The member checks key, name, address, CA:FALSE and the chain to the root. **No member holds a CA key.** Renewal (with 10 of 30 days left) and new memberships wait, with an alert, while the holder is unreachable. Issuance goes through the holder named in the member's state, which leaves room for 0051's other roots. - **Who gets a certificate (`keel.mesh.etcdproof`, after the security review).** The root signs only a request shown to be the member's own. The CSR's own signature is proof of possession of the CSR key (checked before signing); the member's Ed25519 signature over the CSR with its mesh signing key binds that key to the member. A member renews its own address itself (sender = that address's key, proof verified with the trust store's signing key for it). A node being admitted is relayed by its inviter with the admission evidence it signed (names the node, address and signing key; signed by the sender's trusted key); the proof must verify with the evidence's signing key. Evidence naming the inviter's own or any other member's signing key is refused; where the trust store knows the node, the evidence must name the same key; a certificate issued under a signing key the trust store later disowns is revoked at `tend` (`etcdca.impostors`). The holder records the signing key and the relaying member with each certificate, and the relay may revoke it, so `abandoned()` gives a failed join's certificate and learner back. `tests/test_mesh_etcd_root.py` now refuses B for C's address in every form; `tests/etcd_auth_netns.py` does the same on the real cluster. - **etcd auth.** Users are the CNs. | Role | Permission | Granted to | | --- | --- | --- | | `keel-member` | read `/keel/<mesh>/`, write `/keel/<mesh>/etcd/` (the mesh's own keys, e.g. #87's restart lock) | every member | | `keel-vip-<vip>` | read-write `/keel/<mesh>/vip/<vip>/` | the pair's two members only | The holder manages users and roles with an admin certificate (CN `root`) that only it holds (`keel.mesh.etcdauth`). Whether auth is on is read from etcd (`auth status`, cached 5 s), never from a file; every step after `auth enable` in reissue is idempotent, so a crash between them resumes on rerun. `keel vip pair` and `tend` send a pair's record to the holder, which checks it like any node does. Membership changes need root once auth is on, so they now run on the holder: learner promote and remove in `tend`, and member removal inside `revoke`. - **etcdctl instead of the JSON gateway** (`keel.mesh.etcdclient`). The gateway cannot carry a client's CN (etcd's docs, and tested on trixie's 3.5.16). Details: - argv lists, never a shell; - certificates by path; the VIP controller passes its memfds as `/proc/self/fd/N` via `pass_fds`; - no secrets in argv; - env is only `PATH` and `GOMAXPROCS` (fits the controller's TasksMax); - `-w json`, etcd's own timeouts, plus a process kill at +0.5 s; - a failed or unparseable call raises: it is never a renewal. Only etcd's own "requested lease not found" counts as a gone lease. The gateway is now off (`ETCD_ENABLE_GRPC_GATEWAY=false`). keel-vip.service's helper rewrites the controller's memfds when the certificate changes (renewal, reissue). Before this, a controller running past 30 days kept a stale certificate. - **CRL**: still signed with the root key on the holder and distributed as before. Members sign nothing. The 0051 note is in handbook#56. ## Migration of the live layout: `keel mesh etcd reissue` (on the root holder) 1. Checks, changing nothing: every member answers a probe and runs this keel (PKI 2), and every voter is healthy. `--dry-run` stops here and prints the plan. 2. Issues the holder's admin certificate and records its own intermediate. 3. One member at a time, the holder last: `enroll`, then the root signs, then `reissue`. The member drops its intermediate. etcd reads the new certificate at its next handshake, so **nothing restarts**. The next step waits until that member serves the new certificate and every voter is healthy (3 min max, then it stops with the reason; rerun continues). 4. Revokes every intermediate in the root's CRL and pushes the CRL to every member (`crl`). 5. Converges users and roles (including pairs taken from existing VIP claims), runs `auth enable` as root, then reads the mesh as itself to check. **Rollback:** `keel mesh etcd reissue --rollback` disables auth. Certificates stay root-issued: the intermediates are revoked, which is the point. A run that stopped part way needs no rollback: both certificate kinds work together until step 4, and a rerun finishes the job. Upgrade order: upgrade keel and keel-overlay-etcd on every node first, one node at a time (applying the new config restarts etcd once for the gateway change). keel `Breaks: keel-overlay-etcd (<< 0.4.0~)` so etcdctl comes with keel. Then run reissue once. ## Known limits - etcd 3.5 runs no permission check on LeaseKeepAlive. A member outside the pair can keep a VIP lease alive by its ID, which can delay a failover but can never create a second holder (the VIP's checks are unchanged; docs/vip.md). - Rebased on #87, now merged: its multi-endpoint controller and etcd's gate go through etcdctl too. The gate's lock is under `/keel/<mesh>/etcd/`, which every member may write. This is keel 0.22.0. - common#46 (merged) took keel-overlay-etcd 0.3.0; common#48 is 0.4.0, and the Breaks here names it. ## Tests - Unit: recording etcdctl, fake etcd with auth, reissue on a cluster of the old layout including every refusal path, receiver kinds, memfd refresh. Coverage stays above the 95% gate. - `etcd / trixie` and `vip / trixie` now run on etcdctl. - New **`etcd-auth / trixie`** (`tests/test_etcd_auth_netns.py`): 250 ms ±25 ms with 2% loss, measured first. Three members start in the live layout while the VIP is held at A, and reissue runs. It asserts: - one leader and one term, and reads keep working on every member (quorum); - the VIP holder stays A at every 200 ms sample, the ping gap is under 1.5 s, and the epoch is unchanged; - every member is root-issued, every intermediate is revoked everywhere, and auth is on; - C (not in the pair) gets `permission denied` on put, delete, revoke and txn of the VIP keys, while B passes; - no CA key exists on any member, and a CN=root certificate signed with C's own key does not reach etcd; - C's renewal through the root works, waits while A is cut off, and works again afterwards; - rollback turns auth off. It also measures etcdctl's overhead against the controller's 2 s call timeout.

github-advanced-security[bot] (Migrated from github.com) reviewed

2026-10-08 04:39:12 -03:00

keel/mesh/etcdpki.py

CodeQL / Clear-text storage of sensitive information

This expression stores sensitive data (secret) as clear text.

Show more details

## CodeQL / Clear-text storage of sensitive information This expression stores [sensitive data (secret)](1) as clear text. [Show more details](https://github.com/Keel-Linux/keel/security/code-scanning/24)