docs: document how to run the tests in tests/README.md #28

Merged

Nandika-Gupta merged 2 commits from docs/tests-readme into master

2026-10-06 14:02:42 -03:00

Nandika-Gupta commented

2026-10-04 02:46:15 -03:00

(Migrated from github.com)

Closes #26

Adds tests/README.md: what is needed, the commands CI runs, the 100 percent
threshold, the tests that skip without keel, notes on running outside CI
(Linux only; a /tmp with an ACL, as in GitHub Codespaces), and one line per
group of test files. Adds a "Tests" section to README.rst that links to it.

I followed the commands from a fresh checkout in GitHub Codespaces. There
25 tests failed with "writable by group or others": /tmp carries an ACL
that makes new files mode 646 whatever the umask, so the Keel Cloud flag the
tests write is refused. With --basetemp=$HOME/pytmp/run the run gives
924 passed, 7 skipped, coverage 100 percent. The README says so.

Closes #26 Adds `tests/README.md`: what is needed, the commands CI runs, the 100 percent threshold, the tests that skip without keel, notes on running outside CI (Linux only; a `/tmp` with an ACL, as in GitHub Codespaces), and one line per group of test files. Adds a "Tests" section to `README.rst` that links to it. I followed the commands from a fresh checkout in GitHub Codespaces. There 25 tests failed with "writable by group or others": `/tmp` carries an ACL that makes new files mode 646 whatever the umask, so the Keel Cloud flag the tests write is refused. With `--basetemp=$HOME/pytmp/run` the run gives 924 passed, 7 skipped, coverage 100 percent. The README says so.

marcos-mendez (Migrated from github.com) requested changes

2026-10-04 11:33:23 -03:00

marcos-mendez (Migrated from github.com) left a comment

Hi @Nandika-Gupta, thank you, and welcome to Keel Linux!

I followed your instructions on a clean checkout and everything works exactly as written: 924 passed and 7 skipped, coverage at 100% with --fail-under=100 passing, and -k static6 selecting the IPv6 tests. Your facts are right too: CI runs Python 3.13 on ubuntu-latest, the 7 skips are exactly the ones you list for the reasons you give, and every test file is covered in the layout section. Going beyond the issue to explain what conftest.py stubs, and the Codespaces /tmp ACL pitfall, is the kind of detail that saves the next person an hour.

One change before merging, about keeping docs true over time:

Avoid copying facts that live elsewhere. The README repeats the module list from package: in .github/workflows/tests.yml, and the number of skipped tests. When someone adds a module or a skip, the code changes but the README doesn't, and nobody notices because nothing fails. Two common ways out:

Point to the source of truth instead of copying it. For example: "use the module list from package: in .github/workflows/tests.yml", and drop the exact count ("a few tests skip unless a real keel is present; pytest -rs lists them").
Or derive it in the command itself, so it can't drift.

Once that's in and CI is green, I'll merge. Thanks again!

Hi @Nandika-Gupta, thank you, and welcome to Keel Linux! I followed your instructions on a clean checkout and everything works exactly as written: 924 passed and 7 skipped, coverage at 100% with `--fail-under=100` passing, and `-k static6` selecting the IPv6 tests. Your facts are right too: CI runs Python 3.13 on `ubuntu-latest`, the 7 skips are exactly the ones you list for the reasons you give, and every test file is covered in the layout section. Going beyond the issue to explain what `conftest.py` stubs, and the Codespaces `/tmp` ACL pitfall, is the kind of detail that saves the next person an hour. One change before merging, about keeping docs true over time: **Avoid copying facts that live elsewhere.** The README repeats the module list from `package:` in `.github/workflows/tests.yml`, and the number of skipped tests. When someone adds a module or a skip, the code changes but the README doesn't, and nobody notices because nothing fails. Two common ways out: - Point to the source of truth instead of copying it. For example: "use the module list from `package:` in `.github/workflows/tests.yml`", and drop the exact count ("a few tests skip unless a real keel is present; `pytest -rs` lists them"). - Or derive it in the command itself, so it can't drift. Once that's in and CI is green, I'll merge. Thanks again!

Nandika-Gupta commented

2026-10-04 11:51:25 -03:00

(Migrated from github.com)

@marcos-mendez Thanks for the review, that makes sense!
Updated: The command now reads the module list from package: in tests.yml instead of copying it.
I also removed the counts that could go stale (skipped tests, Codespaces failures, the 3.11 note).
Re-ran it on a clean checkout: 924 passed, 7 skipped, coverage 100%.

@marcos-mendez Thanks for the review, that makes sense! Updated: The command now reads the module list from package: in tests.yml instead of copying it. I also removed the counts that could go stale (skipped tests, Codespaces failures, the 3.11 note). Re-ran it on a clean checkout: 924 passed, 7 skipped, coverage 100%.

marcos-mendez (Migrated from github.com) approved these changes

2026-10-06 14:01:30 -03:00

marcos-mendez commented

2026-10-06 16:16:21 -03:00

(Migrated from github.com)

Merged, thanks @Nandika-Gupta! Reading the module list straight from tests.yml was exactly the right fix: the README can't drift from CI now.

Merged, thanks @Nandika-Gupta! Reading the module list straight from `tests.yml` was exactly the right fix: the README can't drift from CI now.

👍 1