Skip to content

ci(python-sdk): build, unit test and e2e workflow against the dev relayer - #661

Draft
HoangDucBach wants to merge 1 commit into
devfrom
test/e2e-python-sdk
Draft

ci(python-sdk): build, unit test and e2e workflow against the dev relayer#661
HoangDucBach wants to merge 1 commit into
devfrom
test/e2e-python-sdk

Conversation

@HoangDucBach

@HoangDucBach HoangDucBach commented Aug 17, 2026

Copy link
Copy Markdown
Collaborator

Ticket

CI for the Python SDK: build from source, unit test, e2e against the dev relayer.

What changed?

New .github/workflows/test-python-sdk.yml, named to sit alongside release-python-sdk.yml:

  • unit — installs from source, runs ruff, then the mocked suite across Python 3.9–3.12. Every pull request touching the package.
  • e2e — runs tests/test_integration.py against the dev relayer on pushes to dev, manual runs, and a weekly cron. Reuses the benchmark-dev environment, so no new secrets are needed.

Also fixes the three pre-existing ruff offences in tests/test_signing.py, which the new lint step would otherwise fail on.

Why is this needed?

The Python SDK had no CI. test.yml only runs the relayer's own e2e script and release-python-sdk.yml just publishes, so nothing built the package or ran either suite on a push — 121 mocked and 23 integration tests sat unexecuted.

What e2e covers

Unauthenticated, runs without any credential:

Group Cases
Health /health returns ok; exposes a version string
Auth rejection unsigned request; signature from a different key; expired timestamp; future timestamp; unregistered key surfaced as MemWalError

Authenticated, needs the environment secrets:

Group Cases
Remember job id and status; remember_and_wait returns blob and owner; default namespace; custom namespace
Recall returns a list; respects limit; result field shape
Analyze fact extraction; owner address
Ask non-empty answer; memories_used
Full flow remember then recall a unique marker in a per-run namespace; remember then ask
Async health, remember, recall, analyze, ask via the async client

The auth-rejection group builds its requests with httpx rather than through the SDK, so it checks the relayer's own contract: timestamp window and signature verification.

Scope

The workflow. Both suites already exist and tests/test_integration.py already reads MEMWAL_SERVER_URL / MEMWAL_PRIVATE_KEY / MEMWAL_ACCOUNT_ID and self-skips its authenticated cases — this wires that up rather than adding tests.

Out of scope

Staging and production environments; tighter namespace isolation for e2e writes.

How was this tested?

  • Unit tests
  • Integration tests
  • End-to-end tests
  • Manual testing
  • Not applicable

Ran what the unit job runs: ruff clean, 116 passed. Workflow validated with actionlint. All four matrix versions pass on this PR, and e2e correctly skips on pull requests.

How can the reviewer verify it?

cd packages/python-sdk-memwal
python -m pip install -e '.[dev]'
python -m ruff check memwal/ tests/
python -m pytest tests/ -m "not integration" -q

docker run --rm -v "$PWD/../..":/repo -w /repo rhysd/actionlint:latest \
  .github/workflows/test-python-sdk.yml

Risks and dependencies

  • Reuses benchmark-dev rather than adding secrets. BENCH_DELEGATE_KEY is the delegate private key the SDK signs with, which test_integration.py reads as MEMWAL_PRIVATE_KEY; BENCH_ACCOUNT_ID maps to MEMWAL_ACCOUNT_ID. Sharing the account is safe because the namespaces do not overlap. BENCH_SERVER_URL falls back to https://relayer.dev.memwal.ai so a missing variable cannot point a run at the wrong relayer. If those secrets turn out not to be set, the unauthenticated group still runs and the rest warn.
  • E2E is skipped on pull requests — environment secrets are withheld from fork PRs, and each authenticated run writes real memories, costing embedding, Walrus storage and gas.
  • Four authenticated tests write into the sdk-test and default namespaces, where benchmark-live.yml uses benchmark — no overlap, but full flow is the only group that isolates itself per run. Worth tightening before this is pointed at production.

Author checklist

  • This pull request maps to one ticket and one logical outcome.
  • I reviewed the complete diff myself.
  • I removed unrelated, debug, and temporary changes.
  • I ran the relevant tests.
  • CI is green.
  • The branch is up to date with its target branch.
  • I added or updated tests where appropriate.
  • I documented any important risk, dependency, rollout, or follow-up.
  • I provided clear verification steps.
  • The pull request is ready for review and is no longer a Draft.

@HoangDucBach
HoangDucBach marked this pull request as draft August 17, 2026 08:37
@HoangDucBach
HoangDucBach force-pushed the test/e2e-python-sdk branch 2 times, most recently from da2a027 to 7aeb92d Compare August 17, 2026 09:40
@HoangDucBach HoangDucBach changed the title ci(python-sdk): build, unit test and e2e workflow across dev/staging/production ci(python-sdk): build, unit test and e2e workflow against the dev relayer Aug 17, 2026
…ayer

The Python SDK had no CI of its own: test.yml only runs the relayer's own
e2e script, and release-python-sdk.yml just publishes. Nothing built the
package from source or ran either test suite on a push.

Add test-python-sdk.yml, named to sit alongside release-python-sdk.yml, with
two jobs:

- `unit` — installs from source and runs ruff plus the mocked suite across
  Python 3.9-3.12, matching requires-python and the pyproject classifiers.
  Runs on every pull request touching the package.
- `e2e` — runs tests/test_integration.py against the dev relayer on pushes to
  dev, manual runs, and a weekly cron. Staging and production are left out
  until their credentials exist.

E2E reuses the `benchmark-dev` environment rather than introducing new
secrets: BENCH_DELEGATE_KEY is the delegate private key the SDK signs with,
which test_integration.py reads as MEMWAL_PRIVATE_KEY. Sharing the account is
safe because the namespaces do not overlap — benchmark-live.yml writes to
`benchmark`, these tests use `sdk-test`, `default` and a per-run
`sdk-e2e-<uuid>`.

E2E is skipped on pull requests: environment secrets are not exposed to fork
PRs, and each authenticated run writes real memories, which costs embedding,
Walrus storage and gas.

The relayer URL falls back to the dev endpoint recorded in
docs/relayer/benchmark-ci-setup.md so a missing BENCH_SERVER_URL cannot
silently point a run at the wrong relayer. Missing credentials only warn —
the authenticated tests self-skip via the `requires_key` marker, so the seven
unauthenticated tests still run: two health checks and five auth-rejection
cases covering unsigned requests, a mismatched signature, expired and future
timestamps, and an unregistered key.

The e2e job name is static on purpose. A `name:` holding an expression is
left unevaluated when the job is skipped, so the check would surface under
two different names and could never back a required status check.

The three pre-existing ruff offences in tests/test_signing.py are fixed here
because the new lint step covers tests/ and would otherwise fail on arrival.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant