fix(tests): keep the suites inside their own temp dirs, and stop them leaking - #477
Merged
Conversation
… leaking Two defects, one class: the test suites reach outside their sandbox. #442 -- THE SUITE MUTATED THE DEVELOPER'S REAL HOME Running the sanctioned hook suite rewrote `~/.claude/settings.json` (both `statusLine.command` and `_codearbiterStatuslineOwner` repointed at whatever plugin root the test process resolved) and littered `~/.codearbiter/` with a ledger, its lock, five session shards, and an update cache. Four modules did it -- test_session_start, test_standup, test_colorlib, test_statusline -- and none of them used `redirect_home`, which already sat beside them in _helpers.py. It broke the maintainer's statusline three times in one day. It makes the suite non-hermetic, so two developers running it end up with different global config. It is why a subagent tripped an "irreversible local destruction" check trying to clean up the mess. And CI never noticed, because a fresh runner has no pre-existing settings to clobber. The fixture is MODULE-level, not per-class, on purpose: the leak is module-wide, those two modules carry 16 and 23 test classes between them, and a per-class `setUp` is one forgotten override away from regressing -- while `setUpModule` covers every class added later for free. It pins ~ AND the two user-global state seams (`CODEARBITER_LEDGER`, `CODEARBITER_UPDATE_STATE`) at one temp dir. #462 -- THE SUITE LEAKED HANDLES AND PROCESSES The same unchanged tree produced `FAILED (errors=2)` on one run and `OK` on the next three. The signal was in the warnings: unclosed `settings.json` and `CONTEXT.md` reads, five never-reaped subprocesses, and an implicitly reclaimed `HTTPError`. On Windows an open handle blocks `TemporaryDirectory` cleanup and a live child holds a temp path, so teardown RAISES instead of the assertion failing -- intermittent ERRORS, not failures, with no code change between runs. Fixed at the source of each: * file reads at test_wire_statusline.py:357/:604 and test_init_codearbiter.py go through a `with` block; * test_taskwrite's real subprocesses are reaped via `addCleanup`, so an assertion firing BEFORE `_finish()` can no longer leave a live child holding the fixture's temp dir; * test_session_start `communicate()`s the spawn handle rather than `wait()`ing it, which drains and closes the pipes; * the `HTTPError` fixture is closed -- constructed with fp=None it still allocates a spooled temp file; * `_MainHarness` stubs BOTH detached spawners. main() ends by firing a real git fetch and a real update-refresh.py, never awaited. Correct in production, wrong in a test: those were the five leaked pids. Both seams exist for injection; the harness now records instead of launching, which also drops test_standup from ~90s to 0.4s. THE GUARD `.github/scripts/test_suite_hermeticity.py`, wired into the hooks job. The obvious guard -- hash the developer's real ~ before and after -- is destructive by observation: it can only detect damage already done, on the machine you care about. Instead each suite runs in a subprocess whose home is a PRISTINE temp directory seeded with a plausible stale-but-real settings.json, and that directory must come back byte-identical: nothing created, modified, or removed. Strictly stronger, and safe to run on the machine that was being damaged. The seeded settings.json is load-bearing -- `heal_statusline_wiring` returns early when no settings file exists, so an empty fake home would hide exactly the write this is about. It deliberately does NOT pin the ledger/update-state env seams. Those are what a well-behaved test uses to point its own state at its own temp dir; pinning them would force correct code to look incorrect and, worse, would let an offender pass by writing to a path the guard itself supplied. A second class asserts the suite emits no ResourceWarning, and a third proves the DETECTOR can fail -- a hermeticity guard that cannot fail is decoration. AC-1/AC-2 the seeded home is byte-identical after the run: no settings rewrite, no ~/.codearbiter/ entries. AC-3 the guard fails if any suite writes outside its temp dir, proven by two deliberate offenders. AC-4 the same audit applied to the .github/scripts suites (hook-guard matrix, codex adapter, dual-host store). Verified red first: the guard reported all 10 leaked paths against the unmodified suite, and 9 ResourceWarnings. Green after: 5/5 guard tests pass, hook suite 1153 tests OK, ci-impact 40 OK. Closes #442. Closes #462.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two defects, one class: the test suites reach outside their sandbox.
#442 — the suite mutated the developer's real home
Running the sanctioned hook suite rewrote
~/.claude/settings.json(bothstatusLine.commandand_codearbiterStatuslineOwnerrepointed at whatever plugin root the test process resolved) and littered~/.codearbiter/with a ledger, its lock, five session shards, and an update cache.Four modules did it —
test_session_start,test_standup,test_colorlib,test_statusline— and none usedredirect_home, which already sat beside them in_helpers.py.It broke the maintainer's statusline three times in one day. It makes the suite non-hermetic, so two developers running it end up with different global config. It is why a subagent tripped an "irreversible local destruction" check trying to clean up the mess. And CI never noticed: a fresh runner has no pre-existing settings to clobber.
The fixture is module-level, not per-class, on purpose. The leak is module-wide, those two modules carry 16 and 23 test classes between them, and a per-class
setUpis one forgotten override away from regressing — whilesetUpModulecovers every class added later for free. It pins~and the two user-global state seams (CODEARBITER_LEDGER,CODEARBITER_UPDATE_STATE) at one temp dir.#462 — the suite leaked handles and processes
The same unchanged tree produced
FAILED (errors=2)on one run andOKon the next three. On Windows an open handle blocksTemporaryDirectorycleanup and a live child holds a temp path, so teardown raises instead of the assertion failing — intermittent errors, not failures, with no code change between runs.test_wire_statusline.py:357/:604,test_init_codearbiter.pywithblocktest_taskwriteaddCleanup, so an assertion firing before_finish()can't leave a live child holding the fixture dirtest_session_startspawn handlecommunicate()rather thanwait()— drains and closes the pipesHTTPErrorfp=Noneit still allocates a spooled temp filemain()fires_MainHarnessThat last one is the interesting one.
session-start.py main()ends by firing a realgit fetchand a realupdate-refresh.py, fully detached and never awaited — correct in production, wrong in a test. Those were the five leaked pids. Both seams exist for injection; the harness now records instead of launching, which also dropstest_standupfrom ~90s to 0.4s.The guard
.github/scripts/test_suite_hermeticity.py, wired into the hooks job.The obvious guard — hash the developer's real
~before and after — is destructive by observation: it can only detect damage already done, on the machine you care about. Instead each suite runs in a subprocess whose home is a pristine temp directory seeded with a plausible stale-but-realsettings.json, and that directory must come back byte-identical: nothing created, modified, or removed. Strictly stronger, and safe to run on the machine that was being damaged.The seeded
settings.jsonis load-bearing:heal_statusline_wiringreturns early when no settings file exists, so an empty fake home would hide exactly the write this issue is about.It deliberately does not pin the ledger/update-state env seams. Those are what a well-behaved test uses to point its own state at its own temp dir; pinning them would force correct code to look incorrect and — worse — would let an offender pass by writing to a path the guard itself supplied.
~/.codearbiter/entries.github/scriptssuites (hook-guard matrix, codex adapter, dual-host store)Plus a class asserting the suite emits no
ResourceWarning, which is what keeps #462 closed rather than merely fixed.Verification
Red first: the guard reported all 10 leaked paths against the unmodified suite, and 9
ResourceWarnings.Green after: 5/5 guard tests pass, hook suite 1153 tests OK,
test_ci_impact40 OK,check_docs_contractOK,sync-core --checkOK.Closes #442. Closes #462.