Remove Mechanical Markdown from examples/ validation#977
Remove Mechanical Markdown from examples/ validation#977
Conversation
Signed-off-by: Sergio Herrera <627709+seherv@users.noreply.github.com>
b92a68e to
65fc9ae
Compare
There was a problem hiding this comment.
Pull request overview
This PR migrates example validation away from Mechanical Markdown to pytest-based integration tests, so the examples/ directory is validated via native Python tests that run the examples and assert on their output.
Changes:
- Replaces the
tox -e examplesMechanical Markdown workflow with a newtox -e integrationpytest environment. - Adds
tests/integration/with aDaprRunnerhelper + one pytest file per example to validate expected output. - Updates project docs (
README.md,AGENTS.md,examples/AGENTS.md) and removesexamples/validate.sh.
Reviewed changes
Copilot reviewed 29 out of 29 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| tox.ini | Removes Mechanical Markdown example envs and adds integration env running pytest. |
| pyproject.toml | Registers the custom example_dir pytest marker. |
| README.md | Updates developer instructions to run tox -e integration. |
| AGENTS.md | Updates repo contributor guidance to refer to pytest integration tests instead of Mechanical Markdown. |
| examples/AGENTS.md | Rewrites examples validation docs to describe pytest/DaprRunner approach. |
| examples/validate.sh | Removes the Mechanical Markdown validation entrypoint script. |
| CLAUDE.md | Adds repository development guidelines. |
| tests/integration/conftest.py | Introduces DaprRunner, output assertion helper, and dapr fixture/marker handling. |
| tests/integration/test_workflow.py | Adds integration coverage for the workflow examples (task chaining, fan-out/fan-in, simple). |
| tests/integration/test_w3c_tracing.py | Adds integration coverage for w3c tracing example with background receiver. |
| tests/integration/test_state_store.py | Adds integration coverage for state store example output. |
| tests/integration/test_state_store_query.py | Adds integration coverage for state store query example with MongoDB setup. |
| tests/integration/test_secret_store.py | Adds integration coverage for secret store example (with/without ACL config). |
| tests/integration/test_pubsub_streaming.py | Adds integration coverage for streaming pubsub example (subscriber/publisher). |
| tests/integration/test_pubsub_streaming_async.py | Adds integration coverage for async streaming pubsub example (subscriber/publisher). |
| tests/integration/test_pubsub_simple.py | Adds integration coverage for basic pubsub example (subscriber/publisher). |
| tests/integration/test_metadata.py | Adds integration coverage for metadata example output. |
| tests/integration/test_langgraph_checkpointer.py | Adds integration coverage for langgraph checkpointer example (skips if deps missing). |
| tests/integration/test_jobs.py | Adds integration coverage for jobs example (management + processing). |
| tests/integration/test_invoke_simple.py | Adds integration coverage for invoke-simple using direct sidecar HTTP call to avoid infinite loop. |
| tests/integration/test_invoke_custom_data.py | Adds integration coverage for invoke-custom-data (receiver + caller). |
| tests/integration/test_invoke_binding.py | Adds integration coverage for invoke-binding with Kafka docker-compose setup. |
| tests/integration/test_grpc_proxying.py | Adds integration coverage for gRPC proxying example (receiver + caller). |
| tests/integration/test_error_handling.py | Adds integration coverage for error handling example output. |
| tests/integration/test_distributed_lock.py | Adds integration coverage for distributed lock example output. |
| tests/integration/test_demo_actor.py | Adds integration coverage for demo actor example (service + client). |
| tests/integration/test_crypto.py | Adds integration coverage for crypto examples with key generation/cleanup fixture. |
| tests/integration/test_conversation.py | Adds integration coverage for conversation examples output. |
| tests/integration/test_configuration.py | Adds integration coverage for configuration example with Redis seeding + update trigger. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| [testenv:integration] | ||
| ; Pytest-based integration tests that validate the examples/ directory. | ||
| ; Usage: tox -e integration # run all | ||
| ; tox -e integration -- test_state_store.py # run one | ||
| passenv = HOME | ||
| basepython = python3 | ||
| changedir = ./examples/ | ||
| deps = | ||
| mechanical-markdown | ||
|
|
||
| changedir = ./tests/integration/ | ||
| commands = | ||
| ./validate.sh conversation | ||
| ./validate.sh crypto | ||
| ./validate.sh metadata | ||
| ./validate.sh error_handling | ||
| ./validate.sh pubsub-simple | ||
| ./validate.sh pubsub-streaming | ||
| ./validate.sh pubsub-streaming-async | ||
| ./validate.sh state_store | ||
| ./validate.sh state_store_query | ||
| ./validate.sh secret_store | ||
| ./validate.sh invoke-simple | ||
| ./validate.sh invoke-custom-data | ||
| ./validate.sh demo_actor | ||
| ./validate.sh invoke-binding | ||
| ./validate.sh grpc_proxying | ||
| ./validate.sh w3c-tracing | ||
| ./validate.sh distributed_lock | ||
| ./validate.sh configuration | ||
| ./validate.sh workflow | ||
| ./validate.sh jobs | ||
| ./validate.sh langgraph-checkpointer | ||
| ./validate.sh ../ | ||
| allowlist_externals=* | ||
|
|
||
| commands_pre = | ||
| pip uninstall -y dapr dapr-ext-grpc dapr-ext-fastapi dapr-ext-langgraph dapr-ext-strands dapr-ext-flask dapr-ext-langgraph dapr-ext-strands | ||
| pip install -e {toxinidir}/ \ | ||
| -e {toxinidir}/ext/dapr-ext-workflow/ \ | ||
| -e {toxinidir}/ext/dapr-ext-grpc/ \ | ||
| -e {toxinidir}/ext/dapr-ext-fastapi/ \ | ||
| -e {toxinidir}/ext/dapr-ext-langgraph/ \ | ||
| -e {toxinidir}/ext/dapr-ext-strands/ \ | ||
| -e {toxinidir}/ext/flask_dapr/ | ||
|
|
||
| [testenv:example-component] | ||
| ; This environment is used to validate a specific example component. | ||
| ; Usage: tox -e example-component -- component_name | ||
| ; Example: tox -e example-component -- conversation | ||
| passenv = HOME | ||
| basepython = python3 | ||
| changedir = ./examples/ | ||
| deps = | ||
| mechanical-markdown | ||
| commands = | ||
| ./validate.sh {posargs} | ||
| pytest {posargs} -v --tb=short |
There was a problem hiding this comment.
The PR renames/removes the examples tox env, but CI still runs tox -e examples (e.g., .github/workflows/validate_examples.yaml:156). As-is, the workflow will fail with “unknown environment” unless the workflow is updated or an alias testenv:examples is kept for backward compatibility.
| from conftest import assert_lines_in_output | ||
|
|
There was a problem hiding this comment.
Importing assert_lines_in_output via from conftest import ... will fail when running pytest from the repo root (because tests/integration/conftest.py isn’t importable as a top-level module). Prefer moving the helper into a regular module (e.g., tests/integration/utils.py) and importing from there, or avoid importing it altogether and keep the assertion helper local to this test file.
| from conftest import assert_lines_in_output | |
| def assert_lines_in_output(output, expected_lines, ordered=False): | |
| if ordered: | |
| start = 0 | |
| for expected_line in expected_lines: | |
| index = output.find(expected_line, start) | |
| assert index != -1, ( | |
| f"Expected line not found in order: {expected_line!r}\n\nOutput was:\n{output}" | |
| ) | |
| start = index + len(expected_line) | |
| else: | |
| for expected_line in expected_lines: | |
| assert expected_line in output, ( | |
| f"Expected line not found: {expected_line!r}\n\nOutput was:\n{output}" | |
| ) |
| def stop(self, proc: subprocess.Popen[str]) -> str: | ||
| """Stop a background process and return its captured output.""" | ||
| self._terminate(proc) | ||
| self._bg = None | ||
| if self._bg_reader is not None: | ||
| self._bg_reader.join(timeout=5) | ||
| self._bg_reader = None | ||
| output = ''.join(self._bg_lines) | ||
| self._bg_lines = [] | ||
| return output |
There was a problem hiding this comment.
stop() assumes the stdout-draining thread has finished, but it only join()s with a timeout. If the thread is still alive after 5s, later self._bg_lines = [] can cause the still-running thread to append into the new list (contaminating subsequent test output) and/or truncate output returned by stop(). Consider joining without a timeout (or verifying the thread exited) before clearing state.
| def cleanup(self) -> None: | ||
| """Stop the background process if still running (teardown safety net).""" | ||
| if self._bg is not None: | ||
| self._terminate(self._bg) | ||
| self._bg = None | ||
| if self._bg_reader is not None: | ||
| self._bg_reader.join(timeout=5) | ||
| self._bg_reader = None | ||
| self._bg_lines = [] | ||
|
|
There was a problem hiding this comment.
cleanup() has the same issue as stop(): it join()s the reader thread with a timeout and then resets internal state. If the reader thread hasn’t actually exited yet, it can append to self._bg_lines after it has been reset. Consider joining without a timeout (or otherwise guaranteeing the reader has stopped) before clearing state.
| data=body, | ||
| headers={'Content-Type': 'application/json'}, | ||
| ) | ||
| urllib.request.urlopen(req) |
There was a problem hiding this comment.
urllib.request.urlopen(req) is called without closing/reading the response body. Using a context manager (and consuming the body) avoids leaking sockets/file descriptors and suppresses ResourceWarning noise in test output, especially if this loop grows.
| urllib.request.urlopen(req) | |
| with urllib.request.urlopen(req) as response: | |
| response.read() |
Description
Removes all references to Mechanical Markdown and replaces them with pytest files that validate the output generated by the examples.
Tests are still flaky and rely on exact log output, which is not ideal. This will be addressed in a future PR.
Issue reference
We strive to have all PR being opened based on an issue, where the problem or feature have been discussed prior to implementation.
Please reference the issue this PR will close: #972 (first iteration, do not close the issue yet)
Checklist
Please make sure you've completed the relevant tasks for this PR, out of the following list: