etcd: a member that the join did not start is refused by every later apply (started marker missing) #188

New issue

Closed

opened 2026-10-16 07:06:49 -03:00 by navigator · 1 comment

navigator commented

2026-10-16 07:06:49 -03:00

Owner

Found in the rehearsal of the cross-site LNMP test, 2026-10-16 (keel 0.23.29, image lnmp 19.0-2, CTs 9101/9102 on BR1 and 9103 on BR2).

The first boot spec had installation.mode: cloud_advanced, overlays.wireguard: disabled and overlays.etcd: disabled (the spec has no network.overlay before the join). The third node joined. The join tried to start etcd and could not:

etcd: not started: /etc/keel/instance.yaml: overlays.etcd: enabled, but etcd requires wireguard, which is disabled
etcd: this node forms with 2 other member(s), but did not start

Then the operator set wireguard: enabled and etcd: enabled and ran keel spec apply --system-only. That apply started etcd, and the cluster has 3 healthy voters. But the apply did not write /var/lib/keel/etcd/started. Each later apply refuses the etcd step and exits with a failure:

overlays.etcd: refused: /var/lib/etcd/default/member holds a member keel never started; the join or keel mesh etcd form removes it when keel-overlay-etcd marked it as etcd-server's own lone member, and an operator any other

Two causes:

keel mesh create and keel mesh join write network.overlay but leave overlays.wireguard: disabled in the spec, so keel.mesh.etcd.start() changes the spec to one that does not validate.
plan_etcd starts etcd when the cluster is known and no member directory is there, but nothing records started after that start.

Workaround: declare wireguard: enabled and etcd: enabled in the first boot spec. Validation accepts it without network.overlay.

Possible fix, least power first: the apply that starts etcd for the cluster of cluster.json writes started; and start() also sets overlays.wireguard: enabled.

Found in the rehearsal of the cross-site LNMP test, 2026-10-16 (keel 0.23.29, image lnmp 19.0-2, CTs 9101/9102 on BR1 and 9103 on BR2). The first boot spec had `installation.mode: cloud_advanced`, `overlays.wireguard: disabled` and `overlays.etcd: disabled` (the spec has no `network.overlay` before the join). The third node joined. The join tried to start etcd and could not: ``` etcd: not started: /etc/keel/instance.yaml: overlays.etcd: enabled, but etcd requires wireguard, which is disabled etcd: this node forms with 2 other member(s), but did not start ``` Then the operator set `wireguard: enabled` and `etcd: enabled` and ran `keel spec apply --system-only`. That apply started etcd, and the cluster has 3 healthy voters. But the apply did not write `/var/lib/keel/etcd/started`. Each later apply refuses the etcd step and exits with a failure: ``` overlays.etcd: refused: /var/lib/etcd/default/member holds a member keel never started; the join or keel mesh etcd form removes it when keel-overlay-etcd marked it as etcd-server's own lone member, and an operator any other ``` Two causes: 1. `keel mesh create` and `keel mesh join` write `network.overlay` but leave `overlays.wireguard: disabled` in the spec, so `keel.mesh.etcd.start()` changes the spec to one that does not validate. 2. `plan_etcd` starts etcd when the cluster is known and no member directory is there, but nothing records `started` after that start. Workaround: declare `wireguard: enabled` and `etcd: enabled` in the first boot spec. Validation accepts it without `network.overlay`. Possible fix, least power first: the apply that starts etcd for the cluster of `cluster.json` writes `started`; and `start()` also sets `overlays.wireguard: enabled`.

navigator referenced this issue from a pull request that will close it,

2026-10-16 07:24:18 -03:00

fix: an apply that starts etcd records it, and the start enables wireguard (keel#188) #191

keel-bridge commented

2026-10-16 10:12:54 -03:00

Member

The bridge mirrored this item to GitHub: https://github.com/Keel-Linux/keel/issues/164
Comments, labels and the open or closed state sync both ways.

The bridge mirrored this item to GitHub: https://github.com/Keel-Linux/keel/issues/164 Comments, labels and the open or closed state sync both ways.

navigator closed this issue

2026-10-16 11:25:06 -03:00