fix: an overlay an upgrade adds to the chain takes its default #88

Merged

marcos-mendez merged 1 commit from fix/overlay-new-in-chain into main

2026-10-08 03:33:45 -03:00

marcos-mendez commented

2026-10-08 03:24:45 -03:00

(Migrated from github.com)

What broke

keel-core 0.1.3 added vip to Core's overlays. After apt full-upgrade (keel 0.20.0, keel-core 0.1.3, keel-overlay-vip 0.1.1), every keel command on two Keel Web nodes failed:

Error: /etc/keel/instance.yaml: overlays.vip: not declared; every overlay of the chain of web is written out, enabled or disabled (decisions 0027, 0041)

Rule 25 held the spec against the chain as it is now, so any overlay added to a chain made every spec written before it invalid.

The fix: option (a)

An overlay of the chain that the spec does not declare, and did not declare when it was last applied, takes the default its manifest gives for installation.mode:

every command that reads the spec warns: overlays.vip: not declared (default: disabled): the chain of web gained it after this spec was last applied, so it takes the manifest's default for simple; write it out
apply --system converges it to that default
diff reports overlays.vip: not declared (default: disabled, observed disabled), or drift when the machine is off the default
inspect already wrote it out from systemd, so a spec written by inspect is complete (now tested)

To tell "the chain gained it" from "the operator dropped it", apply --system records the overlays the spec itself declared (never the filled defaults) in /var/lib/keel/spec/overlays.yaml after a run that did not fail. An overlay in that record that the spec leaves out is still the rule 25 error. With no record (no apply --system since this version), nothing left out is refused. No default is taken without installation.mode, or when the default is ask.

Why (a) and not (b)

0027 makes the spec the operator's document and the source of truth; 0041 rule 25 wants it explicit. Option (b), a maintainer script editing /etc/keel/instance.yaml, would change a file the operator owns, replays elsewhere and may keep under configuration management, without anyone choosing it. Option (a) keeps the spec untouched and keeps the explicit-spec intent for everything the operator wrote, while the emitted spec (inspect, 0039's re-emission) still writes the new overlay out. The record uses the spec at its last apply rather than a manifest version: the manifest carries no per-overlay version, and adding one would raise manifest_version.

Rule 25 needs an amendment for this: Keel-Linux/handbook#54 (proposed, for approval).

Tests

tests/test_spec_new_overlay.py, written first (red on main):

an applied spec without vip keeps validating, applying and diffing after the chain gains vip, with the warning;
validate and diff say "not declared (default: disabled)"; diff is drift when the unit is enabled by hand; an overlay without units says its default in "not compared";
an explicitly declared vip behaves as before (no warning, same (disabled)), and once applied, dropping it is the error;
an overlay the spec declared at its last apply and drops stays an error;
keel inspect writes overlays.vip: disabled;
unit tests for the default per mode, ask, no mode, requires held against defaults, the record's reading, writing and failure to write.

Locally the suite passes except the socket tests in test_mesh_channel.py, test_mesh_vipbridge.py and test_mesh_bridge.py, which fail the same way on main in this sandbox.

Version 0.20.1, with a changelog entry.

## What broke keel-core 0.1.3 added `vip` to Core's overlays. After `apt full-upgrade` (keel 0.20.0, keel-core 0.1.3, keel-overlay-vip 0.1.1), every keel command on two Keel Web nodes failed: `Error: /etc/keel/instance.yaml: overlays.vip: not declared; every overlay of the chain of web is written out, enabled or disabled (decisions 0027, 0041)` Rule 25 held the spec against the chain as it is *now*, so any overlay added to a chain made every spec written before it invalid. ## The fix: option (a) An overlay of the chain that the spec does not declare, **and did not declare when it was last applied**, takes the default its manifest gives for `installation.mode`: - every command that reads the spec warns: `overlays.vip: not declared (default: disabled): the chain of web gained it after this spec was last applied, so it takes the manifest's default for simple; write it out` - `apply --system` converges it to that default - `diff` reports `overlays.vip: not declared (default: disabled, observed disabled)`, or drift when the machine is off the default - `inspect` already wrote it out from systemd, so a spec written by inspect is complete (now tested) To tell "the chain gained it" from "the operator dropped it", `apply --system` records the overlays the spec itself declared (never the filled defaults) in `/var/lib/keel/spec/overlays.yaml` after a run that did not fail. An overlay in that record that the spec leaves out is still the rule 25 error. With no record (no `apply --system` since this version), nothing left out is refused. No default is taken without `installation.mode`, or when the default is `ask`. ## Why (a) and not (b) 0027 makes the spec the operator's document and the source of truth; 0041 rule 25 wants it explicit. Option (b), a maintainer script editing `/etc/keel/instance.yaml`, would change a file the operator owns, replays elsewhere and may keep under configuration management, without anyone choosing it. Option (a) keeps the spec untouched and keeps the explicit-spec intent for everything the operator wrote, while the emitted spec (`inspect`, 0039's re-emission) still writes the new overlay out. The record uses the spec at its last apply rather than a manifest version: the manifest carries no per-overlay version, and adding one would raise `manifest_version`. Rule 25 needs an amendment for this: Keel-Linux/handbook#54 (proposed, for approval). ## Tests `tests/test_spec_new_overlay.py`, written first (red on main): - an applied spec without `vip` keeps validating, applying and diffing after the chain gains `vip`, with the warning; - validate and diff say "not declared (default: disabled)"; diff is drift when the unit is enabled by hand; an overlay without units says its default in "not compared"; - an explicitly declared `vip` behaves as before (no warning, `same (disabled)`), and once applied, dropping it is the error; - an overlay the spec declared at its last apply and drops stays an error; - `keel inspect` writes `overlays.vip: disabled`; - unit tests for the default per mode, `ask`, no mode, requires held against defaults, the record's reading, writing and failure to write. Locally the suite passes except the socket tests in `test_mesh_channel.py`, `test_mesh_vipbridge.py` and `test_mesh_bridge.py`, which fail the same way on `main` in this sandbox. Version 0.20.1, with a changelog entry.