diff --git a/docs/conceptual-guides/index.md b/docs/conceptual-guides/index.md index bf8e422e6e..c8cae07adc 100644 --- a/docs/conceptual-guides/index.md +++ b/docs/conceptual-guides/index.md @@ -23,4 +23,5 @@ package-management package-dependencies make-backend-build components +testing ``` diff --git a/docs/conceptual-guides/testing.md b/docs/conceptual-guides/testing.md new file mode 100644 index 0000000000..035a6ce174 --- /dev/null +++ b/docs/conceptual-guides/testing.md @@ -0,0 +1,131 @@ +--- +myst: + html_meta: + "description": "The kinds of tests a Plone package needs, the two test runners available, and why Plone has testing layers." + "property=og:description": "The kinds of tests a Plone package needs, the two test runners available, and why Plone has testing layers." + "property=og:title": "About testing in Plone" + "keywords": "Plone, testing, pytest, zope.testrunner, testing layers, unittest" +--- + +(about-testing-in-plone)= + +# About testing in Plone + +Testing a Plone package is unlike testing an ordinary Python library, and the difference has one cause. + +An ordinary library can be imported and exercised in microseconds. +A Plone package usually cannot be tested at all until a Plone site exists: a site with your ZCML loaded, your profile installed, and a database behind it. +Building that site takes seconds. + +Everything that follows is a consequence of that one fact. + +## The kinds of tests + +The vocabulary here predates Plone, but Plone uses it in a specific way. + +Unit test +: Exercises a function or class in isolation, with no Plone site at all. + Fast, and worth writing wherever your code allows it. + In practice, much Plone code touches the site so directly that a true unit test is not possible. + +Integration test +: Runs against a real Plone site, in the same process, inside a transaction that is rolled back afterwards. + This is the workhorse. + Most tests you write for an add-on are integration tests. + +Functional test +: Runs against a real Plone site that commits real transactions, so a separate process, such as a browser or an HTTP client, can see the result. + Slower than an integration test, because the isolation is more expensive. + Use it when a request has to arrive over the network. + +Acceptance test +: Drives the whole system as a user would, through a browser. + In Plone this means Robot Framework or a Zope testbrowser for Classic UI. + The Volto frontend has its own browser-based end-to-end testing tools, covered in the Volto documentation. + +The line that matters most in practice is between integration and functional. +It is a question of isolation, and isolation is where the cost is. + +## Why Plone has testing layers + +If building a Plone site takes seconds, and your suite has hundreds of tests, you cannot build one per test. + +A **testing layer** is Plone's answer. +It is an object with two lifecycles: + +- `setUp` and `tearDown` run **once**, and do the expensive work: start Zope, create the site, load ZCML, install your profile. +- `testSetUp` and `testTearDown` run **around every test**, and do only the cheap work needed to keep tests independent. + +That split is the whole idea. +The costly site is built once and shared, while each test still starts from a clean state. + +Isolation happens in the cheap half. +An integration layer opens a transaction before each test and aborts it afterwards, so nothing a test creates survives it. +A functional layer instead stacks a temporary storage on the database, lets the test commit for real, and throws the storage away. + +Layers stack, too. +Your add-on's layer sits on one that made a Plone site, which sits on one that started Zope. +Each is built once, and everything above reuses it. + +You declare your own layer in a `testing.py` module in your package. +That is where you say which ZCML to load and which profile to install. +Both test runners described below consume the same layers. +A layer is not a property of the runner. +Writing one is the shared setup step for either runner, covered in {doc}`/developer-guide/testing/write-a-testing-layer`. + +```{seealso} +The layer machinery lives in the packages that own it: + +- [plone.app.testing](https://github.com/plone/plone.app.testing/blob/master/README.rst): the Plone-specific layers, and the complete reference for the tools used to write a `testing.py`. +- [plone.testing](https://github.com/plone/plone.testing/blob/master/src/plone/testing/README.rst): the underlying layer model. +``` + +## The two test runners + +Python code in the Plone ecosystem is tested with one of two runners. +Both work. +Both use the same layers. +They differ in how you write the tests, not in what the tests can do. + +### zope.testrunner + +The traditional choice, and what Plone core itself uses. + +Tests are `unittest.TestCase` classes. +A class declares the layer it needs by assigning it to a `layer` attribute, and the runner groups tests by layer so each layer is set up once for the whole group. + +That grouping is the runner's defining feature. +It understands layers natively, because layers were built for it. + +### pytest + +The choice most new add-ons make, and the default in packages generated by Cookieplone. + +Tests are plain functions. +What a test needs, it names as an argument, and pytest supplies it: + +```python +def test_portal_title(portal): + assert portal.title == "Plone site" +``` + +pytest has no native concept of a Plone layer. +The [pytest-plone](https://plone.github.io/pytest-plone/) plugin bridges the gap: it takes the layers you already declared in `testing.py` and turns them into pytest fixtures. + +### Choosing + +Neither runner is deprecated, and neither is going away. + +Choose `zope.testrunner` if you contribute to Plone core, or maintain a package whose tests already use it. +There is no reward for converting a working suite. + +Choose `pytest` for new work, unless you have a reason not to. +It is where the wider Python ecosystem is, and it is what Cookieplone gives you. + +The one thing not to do is mix both in the same package. +Two runners means two ways to run the suite and two ways for it to break. + +## Where to go next + +- {doc}`/developer-guide/testing/index`: how to write and run the tests. +``` diff --git a/docs/developer-guide/index.md b/docs/developer-guide/index.md index 22eb2c572f..cad9dc186e 100644 --- a/docs/developer-guide/index.md +++ b/docs/developer-guide/index.md @@ -26,4 +26,5 @@ create-a-distribution standardize-python-project-configuration native-namespace deprecation +testing/index ``` diff --git a/docs/developer-guide/testing/index.md b/docs/developer-guide/testing/index.md new file mode 100644 index 0000000000..d9a3dfef99 --- /dev/null +++ b/docs/developer-guide/testing/index.md @@ -0,0 +1,58 @@ +--- +myst: + html_meta: + "description": "Write and run tests for a Plone backend package, with either zope.testrunner or pytest." + "property=og:description": "Write and run tests for a Plone backend package, with either zope.testrunner or pytest." + "property=og:title": "Test a backend package" + "keywords": "Plone, testing, pytest, zope.testrunner, backend, add-on" +--- + +(testing-backend-packages)= + +# Test a backend package + +How to write and run tests for a Plone package written in Python. + +If you want to understand *why* Plone tests look the way they do before you write any, read {doc}`/conceptual-guides/testing` first. +This page assumes you have. + +```{note} +This covers backend packages. +For testing a Volto add-on, see [Test add-ons](/volto/development/add-ons/test-add-ons-19). +``` + +## Before you start + +You need a testing layer. + +A layer builds the Plone site your tests run against: it loads your ZCML and installs your GenericSetup profile. +You declare it in a `testing.py` module in your package, and a package generated from a Plone template already has one. + +Both test runners consume the same layer. +Choosing a runner does not change how you write `testing.py`. + +If your package has no `testing.py`, write one first. +See {doc}`write-a-testing-layer`. + +## Choose a runner + +| | zope.testrunner | pytest | +| --- | --- | --- | +| Tests are | `unittest.TestCase` classes | plain functions | +| Layers | native | via [pytest-plone](https://plone.github.io/pytest-plone/) | +| Used by | Plone core | most new add-ons, Cookieplone | + +Choose `zope.testrunner` if you contribute to Plone core, or if your package already uses it. +A working test suite is not worth converting. + +Choose pytest for new work. + +Do not use both in one package. + +```{toctree} +:maxdepth: 1 + +write-a-testing-layer +zope-testrunner +pytest +``` diff --git a/docs/developer-guide/testing/pytest.md b/docs/developer-guide/testing/pytest.md new file mode 100644 index 0000000000..5e444a0673 --- /dev/null +++ b/docs/developer-guide/testing/pytest.md @@ -0,0 +1,141 @@ +--- +myst: + html_meta: + "description": "Write and run tests for a Plone package with pytest and pytest-plone." + "property=og:description": "Write and run tests for a Plone package with pytest and pytest-plone." + "property=og:title": "Test with pytest" + "keywords": "Plone, pytest, pytest-plone, fixtures, testing layers" +--- + +(test-with-pytest)= + +# Test with pytest + +This guide shows you how to write and run tests with pytest, the runner most new add-ons use and the default in packages generated by Cookieplone. + +For `zope.testrunner` instead, see {doc}`zope-testrunner`. + +pytest has no native concept of a testing layer. +The [pytest-plone](https://plone.github.io/pytest-plone/) plugin bridges that gap: it turns the layers you declared in your {doc}`testing.py ` into pytest fixtures. +You keep your layers exactly as they are. + +## Install the plugin + +```shell +pip install pytest-plone +``` + +## Write the conftest + +pytest discovers fixtures in a file named `conftest.py`. +Create one at the top of your package. + +Import your testing layers, hand them to `fixtures_factory` with a prefix for each, and inject the result into the module namespace. + +```python +from my.addon.testing import MY_ADDON_FUNCTIONAL_TESTING +from my.addon.testing import MY_ADDON_INTEGRATION_TESTING +from pytest_plone import fixtures_factory + + +pytest_plugins = ["pytest_plone"] + + +globals().update( + fixtures_factory( + ( + (MY_ADDON_FUNCTIONAL_TESTING, "functional"), + (MY_ADDON_INTEGRATION_TESTING, "integration"), + ) + ) +) +``` + +The prefixes name the generated fixtures. +Keep `integration` and `functional`. +The fixtures the plugin provides are built on those names. + +## Write an integration test + +A test is a plain function. +It asks for what it needs by naming it as an argument. + +```python +def test_portal_title(portal): + assert portal.title == "Plone site" +``` + +There is no class, no `setUp`, and no `self`. +`portal` is the Plone site, on the integration layer. + +## Create content without writing setup code + +`pytest-plone` provides a marker that prepares the portal before the test runs. +It applies GenericSetup profiles, creates content, and grants roles. + +```python +import pytest + + +@pytest.mark.portal( + profiles=["my.addon:testing"], + content=[{"type": "Document", "id": "doc1", "title": "A document"}], + roles=["Manager"], +) +def test_document_exists(portal): + assert "doc1" in portal +``` + +As with `zope.testrunner`, no cleanup is needed. +The integration layer aborts the transaction after each test. + +## Write a functional test + +Ask for the functional fixtures instead. +Use them when a request has to arrive over the network, such as a REST API test. + +```python +def test_root_is_public(functional_portal, anon_request): + response = anon_request.get("/") + + assert response.status_code == 200 +``` + +## Run the tests + +```shell +pytest +``` + +Narrow it down while working: + +```shell +pytest -k test_document_exists +pytest tests/test_document.py +``` + +Find out where the time goes: + +```shell +pytest --durations=0 +``` + +## Measure coverage + +The `--cov` options come from the `pytest-cov` plugin, so install it first: + +```shell +pip install pytest-cov +``` + +```shell +pytest --cov=my.addon --cov-report term-missing +``` + +## Where to go next + +`pytest-plone` provides considerably more than shown here: fixtures for add-on install and uninstall checks, content type introspection, vocabularies, authenticated REST API sessions, and class-scoped portals for expensive suites. + +```{seealso} +The [pytest-plone documentation](https://plone.github.io/pytest-plone/) covers the full [fixture reference](https://plone.github.io/pytest-plone/reference/fixtures.html), the marker, and the `fixtures_factory` API. +``` diff --git a/docs/developer-guide/testing/write-a-testing-layer.md b/docs/developer-guide/testing/write-a-testing-layer.md new file mode 100644 index 0000000000..07e776b8c3 --- /dev/null +++ b/docs/developer-guide/testing/write-a-testing-layer.md @@ -0,0 +1,156 @@ +--- +myst: + html_meta: + "description": "Write the testing.py testing layer that both zope.testrunner and pytest-plone build on." + "property=og:description": "Write the testing.py testing layer that both zope.testrunner and pytest-plone build on." + "property=og:title": "Write a testing layer" + "keywords": "Plone, testing layer, testing.py, plone.app.testing, PloneSandboxLayer, PloneWithPackageLayer" +--- + +(write-a-testing-layer)= + +# Write a testing layer + +This guide shows you how to write the testing layer for a Plone add-on. + +The layer is the setup work both test runners share. +Whether you test with {doc}`zope-testrunner` or {doc}`pytest`, the layer is what builds the Plone site your tests run against: it loads your ZCML and installs your GenericSetup profile. +You write it once, in a `testing.py` module in your package. + +For what a layer *is* and why Plone works this way, see {doc}`/conceptual-guides/testing`. +This page is about writing one. + +## Two ways to write it + +`plone.app.testing` gives you two forms. + +Use the **declarative form** for a simple add-on: one package, one profile. +Use the **class form** when setup needs more than that: extra ZCML, several profiles, dependencies loaded first, or setup code that runs against the portal. + +Both produce the same thing—a *fixture* layer—and from that fixture you derive the integration and functional layers your tests name. + +## The declarative form + +For an add-on that loads its own ZCML and installs one profile, instantiate `PloneWithPackageLayer` directly. +No subclass needed. + +```python +from plone.app.testing import FunctionalTesting +from plone.app.testing import IntegrationTesting +from plone.app.testing import PloneWithPackageLayer +from plone.testing.zope import WSGI_SERVER_FIXTURE + +import my.addon + + +MY_ADDON_FIXTURE = PloneWithPackageLayer( + zcml_package=my.addon, + zcml_filename="configure.zcml", + gs_profile_id="my.addon:default", + name="MyAddonFixture", +) + +MY_ADDON_INTEGRATION_TESTING = IntegrationTesting( + bases=(MY_ADDON_FIXTURE,), + name="MyAddonLayer:IntegrationTesting", +) + +MY_ADDON_FUNCTIONAL_TESTING = FunctionalTesting( + bases=(MY_ADDON_FIXTURE, WSGI_SERVER_FIXTURE), + name="MyAddonLayer:FunctionalTesting", +) +``` + +That is a complete `testing.py`. +The three names it exports—the fixture, the integration layer, and the functional layer—are what your tests and your `conftest.py` refer to. + +## The class form + +When the declarative form is not enough, subclass `PloneSandboxLayer` and override two methods. + +```python +from plone.app.testing import applyProfile +from plone.app.testing import FunctionalTesting +from plone.app.testing import IntegrationTesting +from plone.app.testing import PloneSandboxLayer +from plone.testing.zope import WSGI_SERVER_FIXTURE + +import my.addon + + +class MyAddonLayer(PloneSandboxLayer): + + def setUpZope(self, app, configurationContext): + self.loadZCML(package=my.addon) + + def setUpPloneSite(self, portal): + applyProfile(portal, "my.addon:default") + + +MY_ADDON_FIXTURE = MyAddonLayer() + +MY_ADDON_INTEGRATION_TESTING = IntegrationTesting( + bases=(MY_ADDON_FIXTURE,), + name="MyAddonLayer:IntegrationTesting", +) + +MY_ADDON_FUNCTIONAL_TESTING = FunctionalTesting( + bases=(MY_ADDON_FIXTURE, WSGI_SERVER_FIXTURE), + name="MyAddonLayer:FunctionalTesting", +) +``` + +The two methods are the whole difference: + +`setUpZope` +: Runs while Zope starts, before any Plone site exists. + Load your ZCML here. + This is also where you load dependencies—call `self.loadZCML(package=...)` for each, or add their fixtures to the layer's bases. + +`setUpPloneSite` +: Runs against a freshly created Plone site. + Install your profile here with `applyProfile`. + You can install more than one, create shared content, or set roles—anything that should be part of the fixture every test starts from. + +`PloneSandboxLayer` builds on `PLONE_FIXTURE` by default, which is why you get a working Plone site without asking for one. + +## Derive the integration and functional layers + +Both forms end the same way: from the one fixture, derive the layers your tests actually use. + +`IntegrationTesting` wraps each test in a transaction that is rolled back afterwards. +`FunctionalTesting` lets tests commit, which a separate process can then see. + +```python +MY_ADDON_INTEGRATION_TESTING = IntegrationTesting( + bases=(MY_ADDON_FIXTURE,), + name="MyAddonLayer:IntegrationTesting", +) + +MY_ADDON_FUNCTIONAL_TESTING = FunctionalTesting( + bases=(MY_ADDON_FIXTURE, WSGI_SERVER_FIXTURE), + name="MyAddonLayer:FunctionalTesting", +) +``` + +```{important} +Add `WSGI_SERVER_FIXTURE` to the functional layer's bases only when your tests make **real HTTP requests** over a socket—REST API tests, for example, including those that use `pytest-plone`'s `request_factory`. +It starts a WSGI server so an HTTP client can reach the site. +Tests that use a Zope testbrowser do not need it. +``` + +## Use the layers in your tests + +Your `testing.py` is now the single input to whichever runner you use. + +- With `zope.testrunner`, a test class sets `layer = MY_ADDON_INTEGRATION_TESTING`. See {doc}`zope-testrunner`. +- With pytest, you pass the layers to `fixtures_factory` in your `conftest.py`. See {doc}`pytest`. + +## Where to go next + +The two forms shown here cover most add-ons. +For the full set of options—additional Zope products, custom base layers, loading several ZCML files—consult the package that owns the machinery. + +```{seealso} +[plone.app.testing](https://github.com/plone/plone.app.testing/blob/master/README.rst): the complete reference for layers, fixtures, and helpers. +``` diff --git a/docs/developer-guide/testing/zope-testrunner.md b/docs/developer-guide/testing/zope-testrunner.md new file mode 100644 index 0000000000..daaaf16224 --- /dev/null +++ b/docs/developer-guide/testing/zope-testrunner.md @@ -0,0 +1,193 @@ +--- +myst: + html_meta: + "description": "Write and run tests for a Plone package with unittest and zope.testrunner." + "property=og:description": "Write and run tests for a Plone package with unittest and zope.testrunner." + "property=og:title": "Test with zope.testrunner" + "keywords": "Plone, zope.testrunner, unittest, testing layers, coverage" +--- + +(test-with-zope-testrunner)= + +# Test with zope.testrunner + +This guide shows you how to write and run tests with `unittest` and `zope.testrunner`, the runner Plone core uses. + +For pytest instead, see {doc}`pytest`. +This guide assumes your package already has a testing layer. +See {doc}`write-a-testing-layer` if it does not. + +## Install the runner + +```shell +pip install zope.testrunner +``` + +## Write an integration test + +A test is a `unittest.TestCase`. +It declares the layer it needs by assigning it to a `layer` attribute. + +```python +from my.addon.testing import MY_ADDON_INTEGRATION_TESTING +from plone import api + +import unittest + + +class TestDocument(unittest.TestCase): + layer = MY_ADDON_INTEGRATION_TESTING + + def setUp(self): + self.portal = self.layer["portal"] + + def test_portal_title(self): + self.assertEqual(self.portal.title, "Plone site") + + def test_create_document(self): + with api.env.adopt_roles(["Manager"]): + api.content.create( + container=self.portal, + type="Document", + id="doc1", + title="A document", + ) + self.assertIn("doc1", self.portal) +``` + +The `layer` attribute is the whole integration. +The runner reads it, sets that layer up once, and runs every test that declares it. + +`self.layer["portal"]` is how you reach the Plone site. +The layer exposes the objects it built through this mapping. + +Note what `test_create_document` does *not* need: no cleanup. +The integration layer aborts the transaction after each test, so `doc1` is gone before the next test runs. + +## Write a functional test + +Use the functional layer when a request has to arrive over the network, and the site must therefore commit. + +```python +from my.addon.testing import MY_ADDON_FUNCTIONAL_TESTING + +import unittest + + +class TestDocumentView(unittest.TestCase): + layer = MY_ADDON_FUNCTIONAL_TESTING +``` + +Everything else is the same. +Only the layer changes. + +## Run the tests + +```shell +zope-testrunner --test-path src -s my.addon +``` + +`-s` takes the dotted name of the package to search for tests. +`--test-path` points the runner at the directory that holds your source, usually `src`. +This is the form the `tox` environments generated by [plone.meta](https://github.com/plone/meta) use. + +If your package is instead pip-installed in development mode, `--auto-path` locates its directory for you, so you do not have to name a path: + +```shell +zope-testrunner --auto-path -s my.addon +``` + +Add `--auto-color` and `--auto-progress` to either form for readable output: + +```shell +zope-testrunner --auto-color --auto-progress --test-path src -s my.addon +``` + +```{note} +Under Buildout, the equivalent command was `bin/test`. +Buildout generated that script for you. +With a pip-installed Plone, you call `zope-testrunner` directly. +``` + +## Read the output + +The runner groups tests by layer, and tells you so: + +```console +Running my.addon.testing.MyAddon:Integration tests: + Set up my.addon.testing.MyAddon:Integration in 4.521 seconds. + Ran 12 tests with 0 failures, 0 errors and 0 skipped in 0.843 seconds. +Running my.addon.testing.MyAddon:Functional tests: + Set up my.addon.testing.MyAddon:Functional in 0.128 seconds. + Ran 3 tests with 0 failures, 0 errors and 0 skipped in 1.204 seconds. +``` + +This grouping is the runner's defining behavior. +Each layer is set up once, and every test that declared it runs against that one setup. + +The `Set up ... in N seconds` line is the expensive part. +If you see it repeated many times, something is forcing the layer to be rebuilt. + +## Narrow down what runs + +While working on one thing, run one thing. + +Filter by test name, a case-sensitive regular expression: + +```shell +zope-testrunner --auto-path -s my.addon -t test_create_document +``` + +Filter by module, also a regular expression: + +```shell +zope-testrunner --auto-path -s my.addon -m test_document +``` + +Run only unit tests, ignoring every layer: + +```shell +zope-testrunner --auto-path -s my.addon -u +``` + +Run a single layer: + +```shell +zope-testrunner --auto-path -s my.addon --layer Integration +``` + +List what would run, without running it: + +```shell +zope-testrunner --auto-path -s my.addon --list-tests +``` + +`--list-tests` is the fastest way to check that a filter matches what you think it does. + +## Run tests in parallel + +```shell +zope-testrunner --auto-path -s my.addon -j 4 +``` + +Each process sets up its own layers, so this trades memory for wall-clock time. +It pays off on large suites. + +## Measure coverage + +`zope.testrunner` runs as a module, so `coverage` can wrap it: + +```shell +coverage run -m zope.testrunner --auto-path -s my.addon +coverage report +``` + +## Run tests at all levels + +Some tests are registered at a higher level and are skipped by default, notably Robot Framework tests. + +```shell +zope-testrunner --auto-path -s my.addon --all +``` + +Expect this to be considerably slower. diff --git a/styles/config/vocabularies/Plone/accept.txt b/styles/config/vocabularies/Plone/accept.txt index c0b583ac1c..88fad90151 100644 --- a/styles/config/vocabularies/Plone/accept.txt +++ b/styles/config/vocabularies/Plone/accept.txt @@ -16,6 +16,7 @@ Classic UI CMSs CMSUI CommonJS +conftest Cookieplone doctest ETags? @@ -41,6 +42,9 @@ pipx Plate PLIP(s) Plone +plone.app.testing +plone.meta +plone.testing plonecli pluggab(le|ility) pnpm @@ -48,6 +52,8 @@ pnpm prerendered programatically Public UI +pytest +pytest-plone [Qq]uerystring Razzle [Rr]enderers? @@ -55,6 +61,7 @@ RichText Sass Schuko subfolder +testbrowser toggler [Tt]owncrier transpilation @@ -74,3 +81,4 @@ webpack wireframe xkcd Zope +zope.testrunner