Skip to content

fix(tests): keep the suites inside their own temp dirs, and stop them leaking - #477

Merged
SUaDtL merged 2 commits into
mainfrom
fix/442-462-suite-hermeticity
Jul 25, 2026
Merged

fix(tests): keep the suites inside their own temp dirs, and stop them leaking#477
SUaDtL merged 2 commits into
mainfrom
fix/442-462-suite-hermeticity

Conversation

@SUaDtL

@SUaDtL SUaDtL commented Jul 25, 2026

Copy link
Copy Markdown
Collaborator

Two defects, one class: the test suites reach outside their sandbox.

#442 — the suite mutated the developer's real home

Running the sanctioned hook suite rewrote ~/.claude/settings.json (both statusLine.command and _codearbiterStatuslineOwner repointed at whatever plugin root the test process resolved) and littered ~/.codearbiter/ with a ledger, its lock, five session shards, and an update cache.

Four modules did it — test_session_start, test_standup, test_colorlib, test_statusline — and none used redirect_home, which already sat beside them in _helpers.py.

It broke the maintainer's statusline three times in one day. It makes the suite non-hermetic, so two developers running it end up with different global config. It is why a subagent tripped an "irreversible local destruction" check trying to clean up the mess. And CI never noticed: a fresh runner has no pre-existing settings to clobber.

The fixture is module-level, not per-class, on purpose. The leak is module-wide, those two modules carry 16 and 23 test classes between them, and a per-class setUp is one forgotten override away from regressing — while setUpModule covers every class added later for free. It pins ~ and the two user-global state seams (CODEARBITER_LEDGER, CODEARBITER_UPDATE_STATE) at one temp dir.

#462 — the suite leaked handles and processes

The same unchanged tree produced FAILED (errors=2) on one run and OK on the next three. On Windows an open handle blocks TemporaryDirectory cleanup and a live child holds a temp path, so teardown raises instead of the assertion failing — intermittent errors, not failures, with no code change between runs.

leak fix
unclosed reads at test_wire_statusline.py:357/:604, test_init_codearbiter.py routed through a with block
5 never-reaped subprocesses in test_taskwrite reaped via addCleanup, so an assertion firing before _finish() can't leave a live child holding the fixture dir
test_session_start spawn handle communicate() rather than wait() — drains and closes the pipes
implicitly reclaimed HTTPError closed; constructed with fp=None it still allocates a spooled temp file
the two detached spawns main() fires stubbed in _MainHarness

That last one is the interesting one. session-start.py main() ends by firing a real git fetch and a real update-refresh.py, fully detached and never awaited — correct in production, wrong in a test. Those were the five leaked pids. Both seams exist for injection; the harness now records instead of launching, which also drops test_standup from ~90s to 0.4s.

The guard

.github/scripts/test_suite_hermeticity.py, wired into the hooks job.

The obvious guard — hash the developer's real ~ before and after — is destructive by observation: it can only detect damage already done, on the machine you care about. Instead each suite runs in a subprocess whose home is a pristine temp directory seeded with a plausible stale-but-real settings.json, and that directory must come back byte-identical: nothing created, modified, or removed. Strictly stronger, and safe to run on the machine that was being damaged.

The seeded settings.json is load-bearing: heal_statusline_wiring returns early when no settings file exists, so an empty fake home would hide exactly the write this issue is about.

It deliberately does not pin the ledger/update-state env seams. Those are what a well-behaved test uses to point its own state at its own temp dir; pinning them would force correct code to look incorrect and — worse — would let an offender pass by writing to a path the guard itself supplied.

AC covered by
AC-1 / AC-2 the seeded home is byte-identical after the run: no settings rewrite, no ~/.codearbiter/ entries
AC-3 the guard fails if any suite writes outside its temp dir — proven by two deliberate offenders, because a hermeticity guard that cannot fail is decoration
AC-4 the same audit applied to the .github/scripts suites (hook-guard matrix, codex adapter, dual-host store)

Plus a class asserting the suite emits no ResourceWarning, which is what keeps #462 closed rather than merely fixed.

Verification

Red first: the guard reported all 10 leaked paths against the unmodified suite, and 9 ResourceWarnings.

Green after: 5/5 guard tests pass, hook suite 1153 tests OK, test_ci_impact 40 OK, check_docs_contract OK, sync-core --check OK.

Closes #442. Closes #462.

SUaDtL added 2 commits July 25, 2026 18:54
… leaking

Two defects, one class: the test suites reach outside their sandbox.

#442 -- THE SUITE MUTATED THE DEVELOPER'S REAL HOME

Running the sanctioned hook suite rewrote `~/.claude/settings.json` (both
`statusLine.command` and `_codearbiterStatuslineOwner` repointed at whatever
plugin root the test process resolved) and littered `~/.codearbiter/` with a
ledger, its lock, five session shards, and an update cache. Four modules did it
-- test_session_start, test_standup, test_colorlib, test_statusline -- and none
of them used `redirect_home`, which already sat beside them in _helpers.py.

It broke the maintainer's statusline three times in one day. It makes the suite
non-hermetic, so two developers running it end up with different global config.
It is why a subagent tripped an "irreversible local destruction" check trying to
clean up the mess. And CI never noticed, because a fresh runner has no
pre-existing settings to clobber.

The fixture is MODULE-level, not per-class, on purpose: the leak is module-wide,
those two modules carry 16 and 23 test classes between them, and a per-class
`setUp` is one forgotten override away from regressing -- while `setUpModule`
covers every class added later for free. It pins ~ AND the two user-global state
seams (`CODEARBITER_LEDGER`, `CODEARBITER_UPDATE_STATE`) at one temp dir.

#462 -- THE SUITE LEAKED HANDLES AND PROCESSES

The same unchanged tree produced `FAILED (errors=2)` on one run and `OK` on the
next three. The signal was in the warnings: unclosed `settings.json` and
`CONTEXT.md` reads, five never-reaped subprocesses, and an implicitly reclaimed
`HTTPError`. On Windows an open handle blocks `TemporaryDirectory` cleanup and a
live child holds a temp path, so teardown RAISES instead of the assertion
failing -- intermittent ERRORS, not failures, with no code change between runs.

Fixed at the source of each:
  * file reads at test_wire_statusline.py:357/:604 and test_init_codearbiter.py
    go through a `with` block;
  * test_taskwrite's real subprocesses are reaped via `addCleanup`, so an
    assertion firing BEFORE `_finish()` can no longer leave a live child holding
    the fixture's temp dir;
  * test_session_start `communicate()`s the spawn handle rather than `wait()`ing
    it, which drains and closes the pipes;
  * the `HTTPError` fixture is closed -- constructed with fp=None it still
    allocates a spooled temp file;
  * `_MainHarness` stubs BOTH detached spawners. main() ends by firing a real
    git fetch and a real update-refresh.py, never awaited. Correct in
    production, wrong in a test: those were the five leaked pids. Both seams
    exist for injection; the harness now records instead of launching, which
    also drops test_standup from ~90s to 0.4s.

THE GUARD

`.github/scripts/test_suite_hermeticity.py`, wired into the hooks job.

The obvious guard -- hash the developer's real ~ before and after -- is
destructive by observation: it can only detect damage already done, on the
machine you care about. Instead each suite runs in a subprocess whose home is a
PRISTINE temp directory seeded with a plausible stale-but-real settings.json,
and that directory must come back byte-identical: nothing created, modified, or
removed. Strictly stronger, and safe to run on the machine that was being
damaged. The seeded settings.json is load-bearing -- `heal_statusline_wiring`
returns early when no settings file exists, so an empty fake home would hide
exactly the write this is about.

It deliberately does NOT pin the ledger/update-state env seams. Those are what a
well-behaved test uses to point its own state at its own temp dir; pinning them
would force correct code to look incorrect and, worse, would let an offender
pass by writing to a path the guard itself supplied.

A second class asserts the suite emits no ResourceWarning, and a third proves
the DETECTOR can fail -- a hermeticity guard that cannot fail is decoration.

  AC-1/AC-2  the seeded home is byte-identical after the run: no settings
             rewrite, no ~/.codearbiter/ entries.
  AC-3       the guard fails if any suite writes outside its temp dir, proven by
             two deliberate offenders.
  AC-4       the same audit applied to the .github/scripts suites (hook-guard
             matrix, codex adapter, dual-host store).

Verified red first: the guard reported all 10 leaked paths against the
unmodified suite, and 9 ResourceWarnings. Green after: 5/5 guard tests pass,
hook suite 1153 tests OK, ci-impact 40 OK.

Closes #442. Closes #462.
@SUaDtL
SUaDtL merged commit 8a0abe8 into main Jul 25, 2026
42 checks passed
@SUaDtL
SUaDtL deleted the fix/442-462-suite-hermeticity branch July 25, 2026 23:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant