fix: serve the challenge to clients that do not accept gzip #2

Merged

marcos-mendez merged 3 commits from fix/challenge-without-gzip into keel/trixie

2026-10-02 02:10:08 -03:00

marcos-mendez commented

2026-10-02 00:35:19 -03:00

(Migrated from github.com)

Fixes Keel-Linux/keel-web#2: the Anubis challenge returned 500 to clients that do not accept gzip.

Root cause

The cause is in Anubis, not in Coraza (common#25's coraza_delay_response_headers off and the module patch of libnginx-mod-http-coraza#3 are not involved). Upstream 1.27.0's RenderIndex (lib/http.go) refuses one challenged request in 64, at random, when Accept-Encoding does not contain gzip. The refusal goes through respondWithError (HTTP 500), with the log line client was given a challenge but does not in fact support gzip compression. That line is in Anubis's log of the step 7 run, next to the 500.

A missing Accept-Encoding means any coding is acceptable (RFC 9110, 12.5.3), and real clients send none: curl by default, some accessibility and archiving tools, and proxies that strip the header. They got a server error at random and never reached the proof of work, which is the actual defence. Upstream's #1958 (after 1.27.0) only changes the status to the policy's DENY and keeps the sampling, so even with it these clients get an error page instead of the challenge.

Change (1.27.0-0+keel2)

debian/patches/challenge-without-gzip.patch (DEP-3) drops the sampled refusal. It adds TestNonGzipClientGetsChallenge: 4096 requests each without Accept-Encoding, with identity and with br, and all must be challenged. Before the patch it failed at request 67 with status 500.
debian/rules: TestChallengeFor_ErrNotFound, skipped as flaky because of this refusal, runs again.
autopkgtest behind-nginx: 512 requests without Accept-Encoding through nginx must all get the challenge page with 200. The old code passes this by chance about once in 3000 runs.

Evidence

dpkg-buildpackage -b on trixie with golang-1.26: every Go test passes, including the new one. lintian is clean.
On the Keel Web container (step7b image, cloud_simple), with this package:
- 300 curl requests as Googlebot without Accept-Encoding over HTTPS got 300 × 200;
- Chromium sent 200 navigations with Accept-Encoding: identity (checked in Nginx's log) and got 200 × 200, with no 500 and no gzip refusal in Anubis's journal.

Test plan

build / trixie (build plus autopkgtest in the LXC container) green
Released together with Keel-Linux/keel-web's PR, whose keel-web 0.1.1 depends on anubis (>= 1.27.0-0+keel2)

Fixes Keel-Linux/keel-web#2: the Anubis challenge returned 500 to clients that do not accept gzip. ## Root cause The cause is in Anubis, not in Coraza (common#25's `coraza_delay_response_headers off` and the module patch of libnginx-mod-http-coraza#3 are not involved). Upstream 1.27.0's `RenderIndex` (lib/http.go) refuses **one challenged request in 64, at random**, when `Accept-Encoding` does not contain `gzip`. The refusal goes through `respondWithError` (HTTP 500), with the log line `client was given a challenge but does not in fact support gzip compression`. That line is in Anubis's log of the step 7 run, next to the 500. A missing `Accept-Encoding` means any coding is acceptable (RFC 9110, 12.5.3), and real clients send none: curl by default, some accessibility and archiving tools, and proxies that strip the header. They got a server error at random and never reached the proof of work, which is the actual defence. Upstream's #1958 (after 1.27.0) only changes the status to the policy's DENY and keeps the sampling, so even with it these clients get an error page instead of the challenge. ## Change (1.27.0-0+keel2) - `debian/patches/challenge-without-gzip.patch` (DEP-3) drops the sampled refusal. It adds `TestNonGzipClientGetsChallenge`: 4096 requests each without `Accept-Encoding`, with `identity` and with `br`, and all must be challenged. Before the patch it failed at request 67 with status 500. - `debian/rules`: `TestChallengeFor_ErrNotFound`, skipped as flaky because of this refusal, runs again. - autopkgtest `behind-nginx`: 512 requests without `Accept-Encoding` through nginx must all get the challenge page with 200. The old code passes this by chance about once in 3000 runs. ## Evidence - `dpkg-buildpackage -b` on trixie with golang-1.26: every Go test passes, including the new one. lintian is clean. - On the Keel Web container (step7b image, cloud_simple), with this package: - 300 curl requests as Googlebot without `Accept-Encoding` over HTTPS got 300 × 200; - Chromium sent 200 navigations with `Accept-Encoding: identity` (checked in Nginx's log) and got 200 × 200, with no 500 and no gzip refusal in Anubis's journal. ## Test plan - [ ] `build / trixie` (build plus autopkgtest in the LXC container) green - [ ] Released together with Keel-Linux/keel-web's PR, whose keel-web 0.1.1 depends on anubis (>= 1.27.0-0+keel2)