Add a Testing chapter for backend packages

docs/conceptual-guides/index.md

@@ -23,4 +23,5 @@ package-management
 package-dependencies
 make-backend-build
 components
+testing
 ```

docs/conceptual-guides/testing.md

@@ -0,0 +1,131 @@
+---
+myst:
+  html_meta:
+    "description": "The kinds of tests a Plone package needs, the two test runners available, and why Plone has testing layers."
+    "property=og:description": "The kinds of tests a Plone package needs, the two test runners available, and why Plone has testing layers."
+    "property=og:title": "About testing in Plone"
+    "keywords": "Plone, testing, pytest, zope.testrunner, testing layers, unittest"
+---
+
+(about-testing-in-plone)=
+
+# About testing in Plone
+
+Testing a Plone package is unlike testing an ordinary Python library, and the difference has one cause.
+
+An ordinary library can be imported and exercised in microseconds.
+A Plone package usually cannot be tested at all until a Plone site exists: a site with your ZCML loaded, your profile installed, and a database behind it.
+Building that site takes seconds.
+
+Everything that follows is a consequence of that one fact.
+
+## The kinds of tests
+
+The vocabulary here predates Plone, but Plone uses it in a specific way.
+
+Unit test
+:   Exercises a function or class in isolation, with no Plone site at all.
+    Fast, and worth writing wherever your code allows it.
+    In practice, much Plone code touches the site so directly that a true unit test is not possible.
+
+Integration test
+:   Runs against a real Plone site, in the same process, inside a transaction that is rolled back afterwards.
+    This is the workhorse.
+    Most tests you write for an add-on are integration tests.
+
+Functional test
+:   Runs against a real Plone site that commits real transactions, so a separate process, such as a browser or an HTTP client, can see the result.
+    Slower than an integration test, because the isolation is more expensive.
+    Use it when a request has to arrive over the network.
+
+Acceptance test
+:   Drives the whole system as a user would, through a browser.
+    In Plone this means Robot Framework or a Zope testbrowser for Classic UI.
+    The Volto frontend has its own browser-based end-to-end testing tools, covered in the Volto documentation.
+
+The line that matters most in practice is between integration and functional.
+It is a question of isolation, and isolation is where the cost is.
+
+## Why Plone has testing layers
+
+If building a Plone site takes seconds, and your suite has hundreds of tests, you cannot build one per test.
+
+A **testing layer** is Plone's answer.
+It is an object with two lifecycles:
+
+- `setUp` and `tearDown` run **once**, and do the expensive work: start Zope, create the site, load ZCML, install your profile.
+- `testSetUp` and `testTearDown` run **around every test**, and do only the cheap work needed to keep tests independent.
+
+That split is the whole idea.
+The costly site is built once and shared, while each test still starts from a clean state.
+
+Isolation happens in the cheap half.
+An integration layer opens a transaction before each test and aborts it afterwards, so nothing a test creates survives it.
+A functional layer instead stacks a temporary storage on the database, lets the test commit for real, and throws the storage away.
+
+Layers stack, too.
+Your add-on's layer sits on one that made a Plone site, which sits on one that started Zope.
+Each is built once, and everything above reuses it.
+
+You declare your own layer in a `testing.py` module in your package.
+That is where you say which ZCML to load and which profile to install.
+Both test runners described below consume the same layers.
+A layer is not a property of the runner.
+Writing one is the shared setup step for either runner, covered in {doc}`/developer-guide/testing/write-a-testing-layer`.
+
+```{seealso}
+The layer machinery lives in the packages that own it:
+
+- [plone.app.testing](https://github.com/plone/plone.app.testing/blob/master/README.rst): the Plone-specific layers, and the complete reference for the tools used to write a `testing.py`.
+- [plone.testing](https://github.com/plone/plone.testing/blob/master/src/plone/testing/README.rst): the underlying layer model.
+```
+
+## The two test runners
+
+Python code in the Plone ecosystem is tested with one of two runners.
+Both work.
+Both use the same layers.
+They differ in how you write the tests, not in what the tests can do.
+
+### zope.testrunner
+
+The traditional choice, and what Plone core itself uses.
+
+Tests are `unittest.TestCase` classes.
+A class declares the layer it needs by assigning it to a `layer` attribute, and the runner groups tests by layer so each layer is set up once for the whole group.
+
+That grouping is the runner's defining feature.
+It understands layers natively, because layers were built for it.
+
+### pytest
+
+The choice most new add-ons make, and the default in packages generated by Cookieplone.
+
+Tests are plain functions.
+What a test needs, it names as an argument, and pytest supplies it:
+
+```python
+def test_portal_title(portal):
+    assert portal.title == "Plone site"
+```
+
+pytest has no native concept of a Plone layer.
+The [pytest-plone](https://plone.github.io/pytest-plone/) plugin bridges the gap: it takes the layers you already declared in `testing.py` and turns them into pytest fixtures.
+
+### Choosing
+
+Neither runner is deprecated, and neither is going away.
+
+Choose `zope.testrunner` if you contribute to Plone core, or maintain a package whose tests already use it.
+There is no reward for converting a working suite.
+
+Choose `pytest` for new work, unless you have a reason not to.
+It is where the wider Python ecosystem is, and it is what Cookieplone gives you.
+
+The one thing not to do is mix both in the same package.
+Two runners means two ways to run the suite and two ways for it to break.
+
+## Where to go next
+
+- {doc}`/developer-guide/testing/index`: how to write and run the tests.
+```

docs/developer-guide/index.md

@@ -26,4 +26,5 @@ create-a-distribution
 standardize-python-project-configuration
 native-namespace
 deprecation
+testing/index
 ```

docs/developer-guide/testing/index.md

@@ -0,0 +1,58 @@
+---
+myst:
+  html_meta:
+    "description": "Write and run tests for a Plone backend package, with either zope.testrunner or pytest."
+    "property=og:description": "Write and run tests for a Plone backend package, with either zope.testrunner or pytest."
+    "property=og:title": "Test a backend package"
+    "keywords": "Plone, testing, pytest, zope.testrunner, backend, add-on"
+---
+
+(testing-backend-packages)=
+
+# Test a backend package
+
+How to write and run tests for a Plone package written in Python.
+
+If you want to understand *why* Plone tests look the way they do before you write any, read {doc}`/conceptual-guides/testing` first.
+This page assumes you have.
+
+```{note}
+This covers backend packages.
+For testing a Volto add-on, see [Test add-ons](/volto/development/add-ons/test-add-ons-19).
+```
+
+## Before you start
+
+You need a testing layer.
+
+A layer builds the Plone site your tests run against: it loads your ZCML and installs your GenericSetup profile.
+You declare it in a `testing.py` module in your package, and a package generated from a Plone template already has one.
+
+Both test runners consume the same layer.
+Choosing a runner does not change how you write `testing.py`.
+
+If your package has no `testing.py`, write one first.
+See {doc}`write-a-testing-layer`.
+
+## Choose a runner
+
+| | zope.testrunner | pytest |
+| --- | --- | --- |
+| Tests are | `unittest.TestCase` classes | plain functions |
+| Layers | native | via [pytest-plone](https://plone.github.io/pytest-plone/) |
+| Used by | Plone core | most new add-ons, Cookieplone |
+
+Choose `zope.testrunner` if you contribute to Plone core, or if your package already uses it.
+A working test suite is not worth converting.
+
+Choose pytest for new work.
+
+Do not use both in one package.
+
+```{toctree}
+:maxdepth: 1
+
+write-a-testing-layer
+zope-testrunner
+pytest
+```

docs/developer-guide/testing/pytest.md

@@ -0,0 +1,141 @@
+---
+myst:
+  html_meta:
+    "description": "Write and run tests for a Plone package with pytest and pytest-plone."
+    "property=og:description": "Write and run tests for a Plone package with pytest and pytest-plone."
+    "property=og:title": "Test with pytest"
+    "keywords": "Plone, pytest, pytest-plone, fixtures, testing layers"
+---
+
+(test-with-pytest)=
+
+# Test with pytest
+
+This guide shows you how to write and run tests with pytest, the runner most new add-ons use and the default in packages generated by Cookieplone.
+
+For `zope.testrunner` instead, see {doc}`zope-testrunner`.
+
+pytest has no native concept of a testing layer.
+The [pytest-plone](https://plone.github.io/pytest-plone/) plugin bridges that gap: it turns the layers you declared in your {doc}`testing.py <write-a-testing-layer>` into pytest fixtures.
+You keep your layers exactly as they are.
+
+## Install the plugin
+
+```shell
+pip install pytest-plone
+```
+
+## Write the conftest
+
+pytest discovers fixtures in a file named `conftest.py`.
+Create one at the top of your package.
+
+Import your testing layers, hand them to `fixtures_factory` with a prefix for each, and inject the result into the module namespace.
+
+```python
+from my.addon.testing import MY_ADDON_FUNCTIONAL_TESTING
+from my.addon.testing import MY_ADDON_INTEGRATION_TESTING
+from pytest_plone import fixtures_factory
+
+
+pytest_plugins = ["pytest_plone"]
+
+
+globals().update(
+    fixtures_factory(
+        (
+            (MY_ADDON_FUNCTIONAL_TESTING, "functional"),
+            (MY_ADDON_INTEGRATION_TESTING, "integration"),
+        )
+    )
+)
+```
+
+The prefixes name the generated fixtures.
+Keep `integration` and `functional`.
+The fixtures the plugin provides are built on those names.
+
+## Write an integration test
+
+A test is a plain function.
+It asks for what it needs by naming it as an argument.
+
+```python
+def test_portal_title(portal):
+    assert portal.title == "Plone site"
+```
+
+There is no class, no `setUp`, and no `self`.
+`portal` is the Plone site, on the integration layer.
+
+## Create content without writing setup code
+
+`pytest-plone` provides a marker that prepares the portal before the test runs.
+It applies GenericSetup profiles, creates content, and grants roles.
+
+```python
+import pytest
+
+
+@pytest.mark.portal(
+    profiles=["my.addon:testing"],
+    content=[{"type": "Document", "id": "doc1", "title": "A document"}],
+    roles=["Manager"],
+)
+def test_document_exists(portal):
+    assert "doc1" in portal
+```
+
+As with `zope.testrunner`, no cleanup is needed.
+The integration layer aborts the transaction after each test.
+
+## Write a functional test
+
+Ask for the functional fixtures instead.
+Use them when a request has to arrive over the network, such as a REST API test.
+
+```python
+def test_root_is_public(functional_portal, anon_request):
+    response = anon_request.get("/")
+
+    assert response.status_code == 200
+```
+
+## Run the tests
+
+```shell
+pytest
+```
+
+Narrow it down while working:
+
+```shell
+pytest -k test_document_exists
+pytest tests/test_document.py
+```
+
+Find out where the time goes:
+
+```shell
+pytest --durations=0
+```
+
+## Measure coverage
+
+The `--cov` options come from the `pytest-cov` plugin, so install it first:
+
+```shell
+pip install pytest-cov
+```
+
+```shell
+pytest --cov=my.addon --cov-report term-missing
+```
+
+## Where to go next
+
+`pytest-plone` provides considerably more than shown here: fixtures for add-on install and uninstall checks, content type introspection, vocabularies, authenticated REST API sessions, and class-scoped portals for expensive suites.
+
+```{seealso}
+The [pytest-plone documentation](https://plone.github.io/pytest-plone/) covers the full [fixture reference](https://plone.github.io/pytest-plone/reference/fixtures.html), the marker, and the `fixtures_factory` API.
+```