docs/traps.md: the trap of a bats negation that is not the last command #15

New issue

Closed

opened 2026-09-28 05:33:50 -03:00 by marcos-mendez · 0 comments

marcos-mendez commented

2026-09-28 05:33:50 -03:00

(Migrated from github.com)

docs/traps.md is missing the trap found on 2026-09-28, and it is one of the cheapest to fall into again because the broken shape and the working shape are one word apart.

The trap

A bats test body has no assertions of its own: its verdict is the exit status of the last command it ran. Bash does not apply errexit to a negated command — the exemption is in the shell by design, so a script can test a command without dying — so ! cmd on its own line decides the test when it is the final command of the body and is inert everywhere else. Moving a line, or appending one after it, silently changes whether it asserts.

Measured over every default branch in the organization that carries a bats file (19 of them): 57 inert negations in 8 repositories, beside 45 in final position that do assert. The worst was inithooks tests/test-ipconfig.bats, where twelve of the thirteen rejection cases of ipconfig_ip6_syntax were inert — an IPv6 validator's rejection set, in an IPv6-first distribution, a few entries in this same file below the one about IPv6 literals.

Two of the entries already in docs/traps.md had a regression test written for them, and in both cases that test was inert: unit-redis's refutation that the bind fragment never says localhost, and its refutation that no admin user is declared at build time.

What the entry needs

The shape the others use:

Signature — a negative test that passes whatever the code does; deleting the function it refutes leaves the suite green.
Cause — bash does not apply errexit to a negated command, and a bats body's verdict is its last command's status, so only a final-position ! decides anything. Including the shellcheck sweep and the fact that shellcheck grades the two cases apart, error: inert and note: still asserting, which is what makes the sweep exact.
Fix — run ! cmd with bats_require_minimum_version 1.5.0, applied to every negation and not only the inert ones. Plus the two things that bite while converting: run replaces $output/$status/$lines, so a body that refutes and then reads the output of what it was refuting has to keep it in a local first; and run takes a command, not a pipeline.
Enforced — the Negations that assert (SC2314) step of test-shell.yml in the .github repository, which every repository with a bats suite already calls at @main.
Hit — the date and the measurement.

It should also be honest that every predicate turned out to be correct, so nothing shipped broken, and that this is luck rather than evidence — with the one assertion that did fail when it was turned on named, because that one could never have passed for any input.

`docs/traps.md` is missing the trap found on 2026-09-28, and it is one of the cheapest to fall into again because the broken shape and the working shape are one word apart. ## The trap A bats test body has no assertions of its own: its verdict is the exit status of the last command it ran. Bash does not apply errexit to a negated command — the exemption is in the shell by design, so a script can test a command without dying — so `! cmd` on its own line decides the test when it is the final command of the body and is **inert everywhere else**. Moving a line, or appending one after it, silently changes whether it asserts. Measured over every default branch in the organization that carries a bats file (19 of them): **57 inert negations in 8 repositories**, beside 45 in final position that do assert. The worst was `inithooks` `tests/test-ipconfig.bats`, where twelve of the thirteen rejection cases of `ipconfig_ip6_syntax` were inert — an IPv6 validator's rejection set, in an IPv6-first distribution, a few entries in this same file below the one about IPv6 literals. Two of the entries already in `docs/traps.md` had a regression test written for them, and in both cases that test was inert: `unit-redis`'s refutation that the bind fragment never says `localhost`, and its refutation that no `admin` user is declared at build time. ## What the entry needs The shape the others use: - **Signature** — a negative test that passes whatever the code does; deleting the function it refutes leaves the suite green. - **Cause** — bash does not apply errexit to a negated command, and a bats body's verdict is its last command's status, so only a final-position `!` decides anything. Including the shellcheck sweep and the fact that shellcheck grades the two cases apart, `error:` inert and `note:` still asserting, which is what makes the sweep exact. - **Fix** — `run ! cmd` with `bats_require_minimum_version 1.5.0`, applied to every negation and not only the inert ones. Plus the two things that bite while converting: `run` replaces `$output`/`$status`/`$lines`, so a body that refutes and then reads the output of what it was refuting has to keep it in a local first; and `run` takes a command, not a pipeline. - **Enforced** — the `Negations that assert (SC2314)` step of `test-shell.yml` in the `.github` repository, which every repository with a bats suite already calls at `@main`. - **Hit** — the date and the measurement. It should also be honest that every predicate turned out to be correct, so nothing shipped broken, and that this is luck rather than evidence — with the one assertion that did fail when it was turned on named, because that one could never have passed for any input. Related: Keel-Linux/tracker#19 and Keel-Linux/.github#13.