Every negation in the bats suites asserts wherever it stands #16

Merged

marcos-mendez merged 1 commit from test/negations-that-assert into master

2026-09-29 00:58:59 -03:00

marcos-mendez commented

2026-09-28 05:25:44 -03:00

(Migrated from github.com)

Every negation in this suite now asserts wherever it stands. Twenty-two of the thirty-one did not.

Closes #15. Part of Keel-Linux/tracker#19, which has the organization-wide sweep.

Why a bare `!` is inert

A bats test body has no assertions of its own: its verdict is the exit status of the last command it ran. Bash does not apply errexit to a negated command — that exemption is in the shell by design, so a script can test a command without dying — so ! cmd decides the test in final position and is inert everywhere else. The line reads as a refutation either way.

shellcheck grades the two cases apart, which is what made an exact sweep possible:

shellcheck -f gcc --shell=bash tests/*.bats | grep SC2314

error: is an inert one. note: is one in final position, which asserts today and stops asserting the moment a line is added after it.

The headline: twelve of thirteen IPv6 rejection cases

tests/test-ipconfig.bats, "ip6_syntax rejects what is not an IPv6 address": thirteen cases, lines 324–336. Twelve were inert. Only 1:2:3:4:5:6:7::8 decided anything.

Before changing them, each of the thirteen was run against ipconfig_ip6_syntax by hand, so that a green suite afterwards would be evidence rather than a coincidence. All thirteen are refused, and all eight of the acceptance cases are accepted. The validator is correct; it was the tests that were not running.

That matters for what this PR claims: nothing shipped broken here. It is also luck, not evidence, and it is the reason the gate in Keel-Linux/.github exists rather than a note in a review.

The rest

File	Inert	What was not being checked
tests/test-ipconfig.bats	17	the twelve above, plus `valid_config` refusing `bogus` and `''`, `unchanged` refusing an absent stanza, the lxc build leaving no `eth0`, `unchanged` refusing a mismatched inet6 config
tests/test-init-fence.bats	3	`nat_available` for a family with no nat table — the case the predicate exists for — and two processes that must be dead after `stop_mini_server`
tests/test-tagid.bats	2	an untagged page is not tagged, and `@APP_NAME@` does not survive the copy into the writable htdocs

What changed

Every SC2314 site becomes run ! cmd, the nine that were already in final position included, and each file declares bats_require_minimum_version 1.5.0 as the other suites in the organization do. Leaving half of them in the old shape would leave a reader to work out which half is real, and the guard reports severity error only, so a final-position ! is not a failure — it is simply a line that will go inert when somebody appends to the body.

One site was a pipeline and could not be prefixed:

-    ! printf '%s\n' "${lines[@]}" | grep -q WARNING
+    run ! grep -q WARNING <<<"$output"

run takes a command, not a pipeline. The here-string is expanded before run executes, so it reads the output of run "$FENCE" start above it.

Test plan

The thirteen IPv6 rejection cases verified against the validator by hand, before the conversion. All refused.
shellcheck -f gcc --shell=bash --severity=error --include=SC2314 tests/*.bats reports nothing.
Each converted assertion proven to bite: replacing one case with a valid address fails the test at that line.
bats tests/ green, 112 tests. No assertion changed its verdict.

tests/coverage.sh: 99.55 percent of measured lines, threshold 98. Unchanged.

100.00    19/19    firstboot.d/29tagid              ok
100.00     8/8     lib/tagid.sh                     ok
100.00    29/29    firstboot.d/01ipconfig           ok
100.00    72/72    lib/ipconfig.sh                  ok
100.00    28/28    bin/turnkey-init-fence           ok
 98.44    63/64    lib/init-fence.sh                ok
 99.55  total

Tests ship nothing, so package / changelog passes by reporting that nothing that ships changed. No changelog entry and no version bump.

Every negation in this suite now asserts wherever it stands. Twenty-two of the thirty-one did not. Closes #15. Part of Keel-Linux/tracker#19, which has the organization-wide sweep. ## Why a bare `!` is inert A bats test body has no assertions of its own: its verdict is the exit status of the last command it ran. Bash does not apply errexit to a negated command — that exemption is in the shell by design, so a script can test a command without dying — so `! cmd` decides the test in final position and is inert everywhere else. The line reads as a refutation either way. shellcheck grades the two cases apart, which is what made an exact sweep possible: shellcheck -f gcc --shell=bash tests/*.bats | grep SC2314 `error:` is an inert one. `note:` is one in final position, which asserts today and stops asserting the moment a line is added after it. ## The headline: twelve of thirteen IPv6 rejection cases `tests/test-ipconfig.bats`, "ip6_syntax rejects what is not an IPv6 address": thirteen cases, lines 324–336. Twelve were inert. Only `1:2:3:4:5:6:7::8` decided anything. **Before changing them, each of the thirteen was run against `ipconfig_ip6_syntax` by hand**, so that a green suite afterwards would be evidence rather than a coincidence. All thirteen are refused, and all eight of the acceptance cases are accepted. The validator is correct; it was the tests that were not running. That matters for what this PR claims: nothing shipped broken here. It is also luck, not evidence, and it is the reason the gate in `Keel-Linux/.github` exists rather than a note in a review. ## The rest | File | Inert | What was not being checked | |---|---|---| | tests/test-ipconfig.bats | 17 | the twelve above, plus `valid_config` refusing `bogus` and `''`, `unchanged` refusing an absent stanza, the lxc build leaving no `eth0`, `unchanged` refusing a mismatched inet6 config | | tests/test-init-fence.bats | 3 | `nat_available` for a family with no nat table — the case the predicate exists for — and two processes that must be dead after `stop_mini_server` | | tests/test-tagid.bats | 2 | an untagged page is not tagged, and `@APP_NAME@` does not survive the copy into the writable htdocs | ## What changed Every SC2314 site becomes `run ! cmd`, the nine that were already in final position included, and each file declares `bats_require_minimum_version 1.5.0` as the other suites in the organization do. Leaving half of them in the old shape would leave a reader to work out which half is real, and the guard reports severity `error` only, so a final-position `!` is not a failure — it is simply a line that will go inert when somebody appends to the body. One site was a pipeline and could not be prefixed: - ! printf '%s\n' "${lines[@]}" | grep -q WARNING + run ! grep -q WARNING <<<"$output" `run` takes a command, not a pipeline. The here-string is expanded before `run` executes, so it reads the output of `run "$FENCE" start` above it. ## Test plan - [x] The thirteen IPv6 rejection cases verified against the validator by hand, before the conversion. All refused. - [x] `shellcheck -f gcc --shell=bash --severity=error --include=SC2314 tests/*.bats` reports nothing. - [x] Each converted assertion proven to bite: replacing one case with a valid address fails the test at that line. - [x] `bats tests/` green, 112 tests. No assertion changed its verdict. - [x] `tests/coverage.sh`: **99.55 percent** of measured lines, threshold 98. Unchanged. 100.00 19/19 firstboot.d/29tagid ok 100.00 8/8 lib/tagid.sh ok 100.00 29/29 firstboot.d/01ipconfig ok 100.00 72/72 lib/ipconfig.sh ok 100.00 28/28 bin/turnkey-init-fence ok 98.44 63/64 lib/init-fence.sh ok 99.55 total - [x] Tests ship nothing, so `package / changelog` passes by reporting that nothing that ships changed. No changelog entry and no version bump.

marcos-mendez commented

2026-09-29 00:45:53 -03:00

(Migrated from github.com)

Reviewed: all 31 converted refutations go red when the code under test accepts that call (the 13 ip6_syntax cases included), and none reads $output or $status from the run it replaced. One fails with status 3, from the stub, by design. No changes needed. Note: #18 and #14 still carry the old bare negations and will need the same conversion after a merge with master.