Open tools that check claims about AI agents against evidence you pinned: an offline verifier, an observer that records agent runs, and the tools around them. The conformance corpus, vocabulary and admission rules live in https://github.com/probityai/agent-evidence-vectors.
It's for auditors, incident reviewers and relying parties who need a decision they can replay, and for tool authors who want their records checked the same way.
Install the released verifier: uv add agent-evidence-verify==0.5.1, or pip install agent-evidence-verify==0.5.1.
Verified by Probity (github.com/probityai/probity). The probityai command line is released: uv add probityai==0.1.1, or pip install probityai==0.1.1.
Check a sample claim with the released package. You need Python 3.12 or newer and uv; there is nothing to clone:
samples="$(uvx --from agent-evidence-verify==0.5.1 python -c 'import importlib.resources as r; print(r.files("agent_evidence_verify") / "samples")')"
uvx agent-evidence-verify==0.5.1 "$samples/case.json" --policy "$samples/policy-june.json"It prints a decision packet that opens:
Probity decision packet v1
Case: "source-coverage-demo"
Claim: source_text_coverage/v1
Decision: contradicted
Reason: selected_span_missing_from_record
A selected passage is in the source capture but missing from the stored report. Swap in policy-april.json for not_established, or case-complete.json with policy-complete.json for supported.
Two name families, on purpose. agent-evidence-* is the neutral evidence layer: formats and checks any tool can pin without adopting ours. probity* is the product built on it. Each package keeps its own version and release tag, <name>-v<version>.
Evidence layer
| package | registry | version | downloads |
|---|---|---|---|
| agent‑evidence‑mcp | PyPI | 0.2.5 | |
| agent‑evidence‑observer | PyPI | 0.2.0 |
agent-evidence-mcp: offline, read-only MCP server: checks a claim against evidence bytes and replays the conformance vectorsagent-evidence-observer: records what an agent run did as pinned evidence
Product
| package | registry | version | downloads |
|---|---|---|---|
| probityai | PyPI | 0.1.1 | |
| agent‑evidence‑verify | PyPI | 0.5.1 | |
| probity‑verify | PyPI | 0.5.1 | |
| probity‑ratchet | PyPI | 0.4.0 | |
| probity‑ratchet‑cases | PyPI | 0.2.0 | |
| probity‑effects | PyPI | 0.2.0 | - |
| agent‑evidence‑replay | PyPI | 0.1.0 | - |
probityai: theprobityaicommand;probityai verifytodayagent-evidence-verify: offline verifier: supported, contradicted or not establishedprobity-verify: the former name ofagent-evidence-verify; installs itprobity-ratchet: signed, replayable receipts for improving an agent against held-out casesprobity-ratchet-cases: agent task cases graded on final state: a simulated company, tool faults, exact gradersprobity-effects: keeps one effect per agent tool call when its reply is lost, records the trail, tests it against lost-reply scenariosagent-evidence-replay: replays a lab's agent action record and flags a changed or missing step
Install any of them pinned to the version shown: pip install <package>==<version>.
The Claude Code and Cursor plugins ship from harnesses/agent-evidence-plugins.
uv sync --all-packages --all-extras
uv run python -m pytest| page | read it for |
|---|---|
| agent‑evidence‑verify | the verifier, its adapters and its recipes |
| Repository layout | the workspace, the probityai command seam, CI and releases |
| Agent instructions | commands to run before a change |
| Contributing | reporting a problem and opening a pull request |
Apache-2.0, per directory: each member carries its own LICENSE.