unittest: Make more CPython like, add more tests, and fix bugs#1105
Open
tsukasa-au wants to merge 5 commits into
Open
unittest: Make more CPython like, add more tests, and fix bugs#1105tsukasa-au wants to merge 5 commits into
tsukasa-au wants to merge 5 commits into
Conversation
Rather than have this change be pulled into an unrelated commit, I split out all of the changes introduced by running `ruff format` over the unittest codebase. Signed-off-by: Greg Darke <micropython@me.tsukasa.au>
By ensuring that the `setUp`/`tearDown` methods exist in `TestCase`, we simplify the logic needed when subclassing tests. Previously people would need to know if their super class had a `setUp` method before attempting to call it from their overwritten version. Signed-off-by: Greg Darke <micropython@me.tsukasa.au>
Make it possible to run the unittest from within itself. This allows us to write tests that assert: - That the TestSuite had the correct outcomes (failed, errored, skipped, etc) - That the test output the correct information to the user (test names, test outcomes, etc) With this change, we no longer need to rely on raising assertions that are not subclasses of `AssertionError` to show unexpected behaviour in tests. This commit makes a (small) change to how tests for the unittest framework are written. It is now possible to run the entire test framework within itself, allowing us to have assertions against the TestResults and the generated text output. To write assertions against the output TestSuite, we need a way to redirect the output of `print`. This commit adds the `_stdout` variable to the unittest module to allow us to redirect the output. While this is hacky, this variable will go away in a later commit in the series. NOTE: Some of the added tests either had to be skipped or have the incorrect result encoded in them as the existing framework does not behave as expected. These will be fixed in latter commits in the series. Signed-off-by: Greg Darke <micropython@me.tsukasa.au>
NOTE: This code is based on the CPython unittest implementation, and
probably falls under the Python License.
The main things I wanted to accomplish with this were:
- Ensure that each test runs in its own `TestCase` instance
+ This makes it possible to move some per-test setup code from `setUp`
to `__init__`.
- Add `TestCase.run(test_result)`
+ In the CPython implementation, this method gives people a single
place to add custom logic that can run _after_ the tests have
completed, with access to the `unittest.TestResult` instance. This
is useful, as it allows people to only output debug information on
test failures.
While here I ended up making the following changes:
- Moved the test executing code into `TestCase` (and `_Outcome`), from
`_run_suite`.
+ As part of this, I also migrated the exception handling logic to use
`ucontextlib.contextmanager`
- Generate nicer error messages for failures in class setup/teardown
methods.
- Removed `__test_result__` and `__current_test__` hidden variables
- Migrate `TestCase.subTest` to use `ucontextlib.contextmanager`
+ This significantly simplifies the logic
- Make subtest failures output show inline similar to CPython.
Bugs fixed:
- Allow `skip` decorator to work on bare functions
- Stop `expectedFailure` decorator from masking non-assertion failures
- Show correct test name when wrapped with `expectedFailure` decorator
- Show correct test name when TestCase.runTest method exists
- Exceptions in setUp/tearDown should show as test failures
+ Previously they went up the stack, likely causing the process to
exit
- Exceptions in class setUp/tearDown should show as test failures
+ Previously they went up the stack, likely causing the process to
exit
- Don't invoke properties with names that start with `test`
- Non-AssertionError exceptions in subtests show up as "FAIL" instead of
"ERROR".
- Tests now execute in an explicit order
+ This makes it easier to write output tests for the unittest
framework. We _may_ want to explicitly shuffle this order so that
user's don't have cross-test dependencies.
Signed-off-by: Greg Darke <micropython@me.tsukasa.au>
`unittest.main()` will now cause the process to exit with a non-zero status code on test failure. This behaviour is assumed by the test framework used in this repo (in `tools/ci.sh`), is what happens when using the unittest-discover module, and is what CPython does. Removing `unittest/tests/exception.py` as this test always fails (by design). This functionality (a bare function being used as a test raises an error that is not a subclass of `AssertionError`) had a test added in the previous commit (`test_basics.Basics.test_bare_function__error`). Signed-off-by: Greg Darke <micropython@me.tsukasa.au>
Member
|
Thanks for the contribution. And thanks for clearly explaining the reasoning/goals for making these changes.
That is probably a show stopper, since the existing unittest code here is a clean-room implementation with MIT license. Also, we need to keep the For comparison, this PR increases Is there a way to achieve your goals by (1) not copying CPython code, and (2) only modifying the existing |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
NOTE: This code is based on the CPython unittest implementation, and probably falls under the Python License.
The main things I wanted to accomplish with my changes to unittest were:
TestCase.run(test_result), so that users can (easily) wrap their tests, allowing them to output more information in the case of a test failure.test_run_method_overridabletest intest_basics.pyfor an example.TestCaseinstancesetUpto__init__.TestCaseclass hadsetUp,tearDown,setUpClass, andtearDownClassmethodsWhile working through this code I ended up adding more tests, then fixing the issues identified.
As part of the (partial)-rewrite, I ended up using
ucontextlib.contextmanagerin 3 places. I found this helper made it easier to implementunittest.subTest(which was previously implemented with a hand-written context manager), and the_Outcomehelper class (the implementation of which was previously spread acrossSubtestContext,_handle_test_exception, and_run_suite).From what I can see,
ucontextlibis quite small when compiled (it is mostly comments and doc strings).