Skip to content

Let a test kill a mutation of its own module #263

Description

@fniessink

The problem

A @kills registration names the file to mutate and the test that must fail against the mutation. Where the two are the same file, the check cannot run: it reports the mutation as broken and the decorated test fails, whatever the test asserts.

AssertionError: <Outcome.BROKEN: 'broken'> != <Outcome.KILLED: 'killed'> : Mutation(...) was broken:
ModuleNotFoundError: No module named 'tests.update_time.test_visibility.VisibilityTest';
'tests.update_time.test_visibility' is not a package

This is not a rare shape. A check written as a test — the architecture rules, the visibility rules — lives in the module that tests it, so its guard is exactly the case the harness cannot express. It came up twice while refactoring the marker language: for the settings rule in tests/update_time/test_architecture.py and for the private-import rule in tests/update_time/test_visibility.py. Both mutations were run by hand with just mutate instead, which proves the guard once and keeps nothing.

The cause

_run_test installs the mutated source under the module's name and then asks the loader to resolve the test by its dotted name:

sys.modules[_module_name(mutation.path)] = _executed_module(mutated, mutation.path)
unittest.TestLoader().loadTestsFromName(mutation.test)

loadTestsFromName resolves a dotted name by importing the longest prefix it can. When the mutated module is the test's own, the prefix it tries to import runs through that module, which is installed as a plain module object with no __path__, so the import raises rather than backing off to attribute lookup.

The work

Resolve the test against the module object the harness already built, rather than through the loader's import, when the mutated module is the one the test lives in. TestLoader.loadTestsFromName takes a module argument for exactly this: the remainder of the dotted name — the class and the method — is then looked up as attributes of the module handed to it.

What this must not change

  • What each outcome means. STALE and BROKEN keep judging the mutation, KILLED and SURVIVED the test.
  • That the mutation is applied in memory, so an interrupted run leaves no file broken.
  • The modules a run leaves behind: _run_test still restores sys.modules to what it found.
  • Mutating a source module from a test elsewhere, which is how every registration in the suite works today.

Documentation

None. @kills gains no argument and no rule, so nothing a reader of the README or the changelog can observe changes.

Open questions

  1. Should a registration whose file and test disagree about the module keep going through the loader, or should both paths resolve against the executed module?
  2. Is a test that mutates its own module worth a case in tests/test_mutation.py, which today covers a subject module mutated from a test beside it?

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions