Skip to content

Remove Mechanical Markdown from examples/ validation#977

Draft
seherv wants to merge 1 commit intodapr:mainfrom
seherv:remove-mechanical-markdown
Draft

Remove Mechanical Markdown from examples/ validation#977
seherv wants to merge 1 commit intodapr:mainfrom
seherv:remove-mechanical-markdown

Conversation

@seherv
Copy link
Copy Markdown

@seherv seherv commented Apr 10, 2026

Description

Removes all references to Mechanical Markdown and replaces them with pytest files that validate the output generated by the examples.

Tests are still flaky and rely on exact log output, which is not ideal. This will be addressed in a future PR.

Issue reference

We strive to have all PR being opened based on an issue, where the problem or feature have been discussed prior to implementation.

Please reference the issue this PR will close: #972 (first iteration, do not close the issue yet)

Checklist

Please make sure you've completed the relevant tasks for this PR, out of the following list:

  • Code compiles correctly
  • Created/updated tests
  • Extended the documentation

Signed-off-by: Sergio Herrera <627709+seherv@users.noreply.github.com>
@seherv seherv force-pushed the remove-mechanical-markdown branch from b92a68e to 65fc9ae Compare April 10, 2026 13:48
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR migrates example validation away from Mechanical Markdown to pytest-based integration tests, so the examples/ directory is validated via native Python tests that run the examples and assert on their output.

Changes:

  • Replaces the tox -e examples Mechanical Markdown workflow with a new tox -e integration pytest environment.
  • Adds tests/integration/ with a DaprRunner helper + one pytest file per example to validate expected output.
  • Updates project docs (README.md, AGENTS.md, examples/AGENTS.md) and removes examples/validate.sh.

Reviewed changes

Copilot reviewed 29 out of 29 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
tox.ini Removes Mechanical Markdown example envs and adds integration env running pytest.
pyproject.toml Registers the custom example_dir pytest marker.
README.md Updates developer instructions to run tox -e integration.
AGENTS.md Updates repo contributor guidance to refer to pytest integration tests instead of Mechanical Markdown.
examples/AGENTS.md Rewrites examples validation docs to describe pytest/DaprRunner approach.
examples/validate.sh Removes the Mechanical Markdown validation entrypoint script.
CLAUDE.md Adds repository development guidelines.
tests/integration/conftest.py Introduces DaprRunner, output assertion helper, and dapr fixture/marker handling.
tests/integration/test_workflow.py Adds integration coverage for the workflow examples (task chaining, fan-out/fan-in, simple).
tests/integration/test_w3c_tracing.py Adds integration coverage for w3c tracing example with background receiver.
tests/integration/test_state_store.py Adds integration coverage for state store example output.
tests/integration/test_state_store_query.py Adds integration coverage for state store query example with MongoDB setup.
tests/integration/test_secret_store.py Adds integration coverage for secret store example (with/without ACL config).
tests/integration/test_pubsub_streaming.py Adds integration coverage for streaming pubsub example (subscriber/publisher).
tests/integration/test_pubsub_streaming_async.py Adds integration coverage for async streaming pubsub example (subscriber/publisher).
tests/integration/test_pubsub_simple.py Adds integration coverage for basic pubsub example (subscriber/publisher).
tests/integration/test_metadata.py Adds integration coverage for metadata example output.
tests/integration/test_langgraph_checkpointer.py Adds integration coverage for langgraph checkpointer example (skips if deps missing).
tests/integration/test_jobs.py Adds integration coverage for jobs example (management + processing).
tests/integration/test_invoke_simple.py Adds integration coverage for invoke-simple using direct sidecar HTTP call to avoid infinite loop.
tests/integration/test_invoke_custom_data.py Adds integration coverage for invoke-custom-data (receiver + caller).
tests/integration/test_invoke_binding.py Adds integration coverage for invoke-binding with Kafka docker-compose setup.
tests/integration/test_grpc_proxying.py Adds integration coverage for gRPC proxying example (receiver + caller).
tests/integration/test_error_handling.py Adds integration coverage for error handling example output.
tests/integration/test_distributed_lock.py Adds integration coverage for distributed lock example output.
tests/integration/test_demo_actor.py Adds integration coverage for demo actor example (service + client).
tests/integration/test_crypto.py Adds integration coverage for crypto examples with key generation/cleanup fixture.
tests/integration/test_conversation.py Adds integration coverage for conversation examples output.
tests/integration/test_configuration.py Adds integration coverage for configuration example with Redis seeding + update trigger.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +41 to +49
[testenv:integration]
; Pytest-based integration tests that validate the examples/ directory.
; Usage: tox -e integration # run all
; tox -e integration -- test_state_store.py # run one
passenv = HOME
basepython = python3
changedir = ./examples/
deps =
mechanical-markdown

changedir = ./tests/integration/
commands =
./validate.sh conversation
./validate.sh crypto
./validate.sh metadata
./validate.sh error_handling
./validate.sh pubsub-simple
./validate.sh pubsub-streaming
./validate.sh pubsub-streaming-async
./validate.sh state_store
./validate.sh state_store_query
./validate.sh secret_store
./validate.sh invoke-simple
./validate.sh invoke-custom-data
./validate.sh demo_actor
./validate.sh invoke-binding
./validate.sh grpc_proxying
./validate.sh w3c-tracing
./validate.sh distributed_lock
./validate.sh configuration
./validate.sh workflow
./validate.sh jobs
./validate.sh langgraph-checkpointer
./validate.sh ../
allowlist_externals=*

commands_pre =
pip uninstall -y dapr dapr-ext-grpc dapr-ext-fastapi dapr-ext-langgraph dapr-ext-strands dapr-ext-flask dapr-ext-langgraph dapr-ext-strands
pip install -e {toxinidir}/ \
-e {toxinidir}/ext/dapr-ext-workflow/ \
-e {toxinidir}/ext/dapr-ext-grpc/ \
-e {toxinidir}/ext/dapr-ext-fastapi/ \
-e {toxinidir}/ext/dapr-ext-langgraph/ \
-e {toxinidir}/ext/dapr-ext-strands/ \
-e {toxinidir}/ext/flask_dapr/

[testenv:example-component]
; This environment is used to validate a specific example component.
; Usage: tox -e example-component -- component_name
; Example: tox -e example-component -- conversation
passenv = HOME
basepython = python3
changedir = ./examples/
deps =
mechanical-markdown
commands =
./validate.sh {posargs}
pytest {posargs} -v --tb=short
Copy link

Copilot AI Apr 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR renames/removes the examples tox env, but CI still runs tox -e examples (e.g., .github/workflows/validate_examples.yaml:156). As-is, the workflow will fail with “unknown environment” unless the workflow is updated or an alias testenv:examples is kept for backward compatibility.

Copilot uses AI. Check for mistakes.
Comment on lines +3 to +4
from conftest import assert_lines_in_output

Copy link

Copilot AI Apr 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Importing assert_lines_in_output via from conftest import ... will fail when running pytest from the repo root (because tests/integration/conftest.py isn’t importable as a top-level module). Prefer moving the helper into a regular module (e.g., tests/integration/utils.py) and importing from there, or avoid importing it altogether and keep the assertion helper local to this test file.

Suggested change
from conftest import assert_lines_in_output
def assert_lines_in_output(output, expected_lines, ordered=False):
if ordered:
start = 0
for expected_line in expected_lines:
index = output.find(expected_line, start)
assert index != -1, (
f"Expected line not found in order: {expected_line!r}\n\nOutput was:\n{output}"
)
start = index + len(expected_line)
else:
for expected_line in expected_lines:
assert expected_line in output, (
f"Expected line not found: {expected_line!r}\n\nOutput was:\n{output}"
)

Copilot uses AI. Check for mistakes.
Comment on lines +100 to +109
def stop(self, proc: subprocess.Popen[str]) -> str:
"""Stop a background process and return its captured output."""
self._terminate(proc)
self._bg = None
if self._bg_reader is not None:
self._bg_reader.join(timeout=5)
self._bg_reader = None
output = ''.join(self._bg_lines)
self._bg_lines = []
return output
Copy link

Copilot AI Apr 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

stop() assumes the stdout-draining thread has finished, but it only join()s with a timeout. If the thread is still alive after 5s, later self._bg_lines = [] can cause the still-running thread to append into the new list (contaminating subsequent test output) and/or truncate output returned by stop(). Consider joining without a timeout (or verifying the thread exited) before clearing state.

Copilot uses AI. Check for mistakes.
Comment on lines +111 to +120
def cleanup(self) -> None:
"""Stop the background process if still running (teardown safety net)."""
if self._bg is not None:
self._terminate(self._bg)
self._bg = None
if self._bg_reader is not None:
self._bg_reader.join(timeout=5)
self._bg_reader = None
self._bg_lines = []

Copy link

Copilot AI Apr 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cleanup() has the same issue as stop(): it join()s the reader thread with a timeout and then resets internal state. If the reader thread hasn’t actually exited yet, it can append to self._bg_lines after it has been reset. Consider joining without a timeout (or otherwise guaranteeing the reader has stopped) before clearing state.

Copilot uses AI. Check for mistakes.
data=body,
headers={'Content-Type': 'application/json'},
)
urllib.request.urlopen(req)
Copy link

Copilot AI Apr 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

urllib.request.urlopen(req) is called without closing/reading the response body. Using a context manager (and consuming the body) avoids leaking sockets/file descriptors and suppresses ResourceWarning noise in test output, especially if this loop grows.

Suggested change
urllib.request.urlopen(req)
with urllib.request.urlopen(req) as response:
response.read()

Copilot uses AI. Check for mistakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

refactor: mv mechanical markdown tests to pytest native

2 participants