feat: the Keel Web overlay packages, nginx, coraza and anubis (0041 step 6) #21

Merged

marcos-mendez merged 9 commits from feat/web-overlay-packages into 19.x

2026-10-01 14:38:09 -03:00

marcos-mendez commented

2026-10-01 00:53:28 -03:00

(Migrated from github.com)

Step 6 of handbook decision 0041's first implementation: keel-overlay-nginx, keel-overlay-coraza and keel-overlay-anubis, a source package each under packages/, with their manifests. Design and reasons in packages/README.md, "The Web overlays".

Done criterion, on Core

Measured on an LXC container (s6-test) made from the step 4 Core image debian-13-keel-core_19.0-3+step4-20260930_amd64.tar.zst, headless first boot, keel upgraded to 0.14.0 (main, 527dc85), the three overlays installed in one apt transaction with the step 5 packages built with gbp buildpackage from their keel/trixie heads (libcoraza 2dd7785, coreruleset 7411b7a, libnginx-mod-http-coraza 630c68f, anubis e2512d3).

nginx answers /keel-health with 204 on the loopback only

[::1]          -> 204
127.0.0.1      -> 204
10.0.3.45 -> 404 (Debian's default site: / -> 200)
fd42:b2:0:1:77a4:e265:757e:4ecd -> 404 (Debian's default site: / -> 200)

Coraza and Anubis installed and disabled

modules-enabled: 50-mod-stream.conf 90-keel-streams.conf
worker 5492..5495: no libcoraza
waf probe -> 204 (not blocked)
anubis@keel.service: disabled inactive
/etc/anubis/keel.key: No such file or directory
[::1]:8923 -> 000
keel manifest validate: every manifest ok, core resolved

Turned on in the spec, waf-blocks and anubis pass. A spec with appliance: web (the format's web.yaml placed by hand until step 7 ships keel-web), coraza: enabled, anubis: enabled, applied with keel spec apply --system-only:

keel enabled and started anubis@keel.service; the key unit made /etc/anubis/keel.key (0600 root, 64 hex) on that start; Monit keel-check-anubis OK.
keel wrote waf-blocks into Monit's file but loaded nothing: Coraza has no unit. Monit: failed protocol test ... Server returned status 204.
/usr/lib/keel/overlays/coraza/state enabled: coraza: enabled: the probe got 403 and /keel-health 204. Next Monit run: 'keel-check-waf-blocks' connection succeeded, summary OK for nginx, waf-blocks and anubis.

tests/overlay-web.bats on the same container: 28 of 28.

Fact 1 (corazawaf/coraza-nginx#139)

Handled in the overlay's enable hook, /usr/lib/keel/overlays/coraza/state enabled: links, nginx -t, reload, then waits for the probe to get 403 and /keel-health 204; otherwise restores the links, reloads, checks Nginx answers, and fails naming the cause. Reproduced and covered by a test: SecRule ARGS "@rx (" ... appended to REQUEST-900 passes nginx -t, all workers exit with fatal code 2; the hook rolls back and Nginx answers 204 again. keel 0.14.0 does not call the hook yet: listed for step 7 / the 0042 work in the README.

What keel lacks

Running /usr/lib/keel/overlays/<name>/state enabled|disabled on a state change (Coraza has no unit to converge).
diff/inspect for a unit-less overlay ("not compared" today).
Secrets: generate: true yields a short token for the conf, not an Anubis key; the emitter should write {file: /etc/anubis/keel.key} and a replica should get the primary's key before its first start.
Monit re-encodes the waf-blocks path (%253C); CRS still blocks it (403).

Notes

Anubis's manifest departs from manifest-v1.md's example: unit: anubis@keel.service (step 5 ships a template) and address: "::1". A handbook erratum should follow.
[::1] rather than a unix socket for Anubis: the manifest's check is a TCP probe and format v1 has no socket check.
keel-health.conf listens on [::1]:80/127.0.0.1:80, so it answers all loopback traffic on port 80 whatever the Host.
CrowdSec's default crowdsecurity/whitelists on the Core image already whitelists 127.0.0.1 and ::1.
CI: the Web overlays build and lint in packages / build; their live suite needs systemd, which Keel runs in system containers only, so it ran on the test VM. install now installs only the Core four. The two hook scripts are unit tested at 100 percent in tests / coverage.

After review

Coraza hook: a failed reload rolls back; it probes only after the pre-reload workers stop taking requests (an old worker that already had CRS answered 403 while the new ones died); flock serialises runs.
Upgrade guard: a dpkg trigger on coreruleset, the module and libcoraza runs state recheck, which turns Coraza off when the new files kill the workers (tested with a coreruleset carrying a bad rule). Other reloads (by hand, cert renewal, other modules' trigger, edits under /etc/coreruleset) are documented as unguarded.
prerm remove drops both links; a purge of the Anubis overlay stops and disables anubis@keel.
Anubis key: links the spec's {file: PATH}, refuses a missing one, and makes none on a database replica or where /etc/anubis/keel.key.from-primary exists. keel's part: Keel-Linux/keel#62. Manifest erratum: Keel-Linux/handbook#33.
19.x merged in (LXC CI); new job web / trixie runs tests/overlay-web.bats (32 tests) in a booted LXC system container, with step 5's packages built there.

The first live run was on edfcf68. cfedd9a only joins the hook's continued lines (kcov measured 82 of 84 on the CI runner); its unit suite passes unchanged and coverage is 100.

The container was destroyed after the run.

Step 6 of handbook decision 0041's first implementation: `keel-overlay-nginx`, `keel-overlay-coraza` and `keel-overlay-anubis`, a source package each under `packages/`, with their manifests. Design and reasons in `packages/README.md`, "The Web overlays". ## Done criterion, on Core Measured on an LXC container (`s6-test`) made from the step 4 Core image `debian-13-keel-core_19.0-3+step4-20260930_amd64.tar.zst`, headless first boot, keel upgraded to 0.14.0 (main, 527dc85), the three overlays installed in one apt transaction with the step 5 packages built with `gbp buildpackage` from their `keel/trixie` heads (libcoraza 2dd7785, coreruleset 7411b7a, libnginx-mod-http-coraza 630c68f, anubis e2512d3). **nginx answers `/keel-health` with 204 on the loopback only** ``` [::1] -> 204 127.0.0.1 -> 204 10.0.3.45 -> 404 (Debian's default site: / -> 200) fd42:b2:0:1:77a4:e265:757e:4ecd -> 404 (Debian's default site: / -> 200) ``` **Coraza and Anubis installed and disabled** ``` modules-enabled: 50-mod-stream.conf 90-keel-streams.conf worker 5492..5495: no libcoraza waf probe -> 204 (not blocked) anubis@keel.service: disabled inactive /etc/anubis/keel.key: No such file or directory [::1]:8923 -> 000 keel manifest validate: every manifest ok, core resolved ``` **Turned on in the spec, `waf-blocks` and `anubis` pass.** A spec with `appliance: web` (the format's `web.yaml` placed by hand until step 7 ships `keel-web`), `coraza: enabled`, `anubis: enabled`, applied with `keel spec apply --system-only`: - keel enabled and started `anubis@keel.service`; the key unit made `/etc/anubis/keel.key` (0600 root, 64 hex) on that start; Monit `keel-check-anubis` OK. - keel wrote `waf-blocks` into Monit's file but loaded nothing: Coraza has no unit. Monit: `failed protocol test ... Server returned status 204`. - `/usr/lib/keel/overlays/coraza/state enabled`: `coraza: enabled: the probe got 403 and /keel-health 204`. Next Monit run: `'keel-check-waf-blocks' connection succeeded`, summary OK for nginx, waf-blocks and anubis. `tests/overlay-web.bats` on the same container: 28 of 28. ## Fact 1 (corazawaf/coraza-nginx#139) Handled in the overlay's enable hook, `/usr/lib/keel/overlays/coraza/state enabled`: links, `nginx -t`, reload, then waits for the probe to get 403 and `/keel-health` 204; otherwise restores the links, reloads, checks Nginx answers, and fails naming the cause. Reproduced and covered by a test: `SecRule ARGS "@rx (" ...` appended to `REQUEST-900` passes `nginx -t`, all workers exit with fatal code 2; the hook rolls back and Nginx answers 204 again. keel 0.14.0 does not call the hook yet: listed for step 7 / the 0042 work in the README. ## What keel lacks 1. Running `/usr/lib/keel/overlays/<name>/state enabled|disabled` on a state change (Coraza has no unit to converge). 2. `diff`/`inspect` for a unit-less overlay ("not compared" today). 3. Secrets: `generate: true` yields a short token for the conf, not an Anubis key; the emitter should write `{file: /etc/anubis/keel.key}` and a replica should get the primary's key before its first start. 4. Monit re-encodes the waf-blocks path (`%253C`); CRS still blocks it (403). ## Notes - Anubis's manifest departs from manifest-v1.md's example: `unit: anubis@keel.service` (step 5 ships a template) and `address: "::1"`. A handbook erratum should follow. - `[::1]` rather than a unix socket for Anubis: the manifest's check is a TCP probe and format v1 has no socket check. - `keel-health.conf` listens on `[::1]:80`/`127.0.0.1:80`, so it answers all loopback traffic on port 80 whatever the Host. - CrowdSec's default `crowdsecurity/whitelists` on the Core image already whitelists `127.0.0.1` and `::1`. - CI: the Web overlays build and lint in `packages / build`; their live suite needs systemd, which Keel runs in system containers only, so it ran on the test VM. `install` now installs only the Core four. The two hook scripts are unit tested at 100 percent in `tests / coverage`. ## After review - Coraza hook: a failed reload rolls back; it probes only after the pre-reload workers stop taking requests (an old worker that already had CRS answered 403 while the new ones died); `flock` serialises runs. - Upgrade guard: a dpkg trigger on coreruleset, the module and libcoraza runs `state recheck`, which turns Coraza off when the new files kill the workers (tested with a coreruleset carrying a bad rule). Other reloads (by hand, cert renewal, other modules' trigger, edits under /etc/coreruleset) are documented as unguarded. - `prerm remove` drops both links; a purge of the Anubis overlay stops and disables `anubis@keel`. - Anubis key: links the spec's `{file: PATH}`, refuses a missing one, and makes none on a database replica or where `/etc/anubis/keel.key.from-primary` exists. keel's part: Keel-Linux/keel#62. Manifest erratum: Keel-Linux/handbook#33. - 19.x merged in (LXC CI); new job `web / trixie` runs `tests/overlay-web.bats` (32 tests) in a booted LXC system container, with step 5's packages built there. The first live run was on edfcf68. cfedd9a only joins the hook's continued lines (kcov measured 82 of 84 on the CI runner); its unit suite passes unchanged and coverage is 100. The container was destroyed after the run.