Skip to content

Add the monitoring/kpi_gate skill bundle implementing the issue #317 interface - #318

Open
mrmasa88 wants to merge 2 commits into
ARPAHLS:mainfrom
0x-AO-Protocol:feat/issue-317-kpi-gate
Open

Add the monitoring/kpi_gate skill bundle implementing the issue #317 interface#318
mrmasa88 wants to merge 2 commits into
ARPAHLS:mainfrom
0x-AO-Protocol:feat/issue-317-kpi-gate

Conversation

@mrmasa88

Copy link
Copy Markdown
Contributor

Description

Adds the monitoring/kpi_gate registry skill implementing the interface agreed in #317 (three input JSON Schemas, output contract, four-stage fail-closed validation order, agent-loop contract, and the end-to-end example). Design rationale and the frozen interface live in the #317 comments and are not re-argued here.

Acceptance criteria mapped to the diff:

  • Three-state vocabulary (error / warning / insufficient_data with reason codes) — skills/monitoring/kpi_gate/skill.py; branch coverage in test_skill.py (28 offline tests)
  • Strict schemas, closed rule set, fail-closed errors — stdlib validation mirroring the three [New Skill]: monitoring/funnel_monitor — deterministic funnel health evaluation against charter thresholds #317 schemas (shipped verbatim under schemas/ as reference documents; requirements: [] stays empty by design, so no runtime jsonschema dependency); closed error registry INVALID_METRICS_SCHEMA, INVALID_POLICY_SCHEMA, INVALID_BENCHMARKS_SCHEMA, NO_METRICS_PROVIDED, UNKNOWN_METRIC_KEY, UNKNOWN_RULE_METRIC, UNKNOWN_DENOMINATOR_METRIC, BENCHMARK_VERSION_MISSING, BENCHMARK_REF_UNRESOLVED — one dedicated test per code, including the dependentRequired rejection
  • End-to-end example — the [New Skill]: monitoring/funnel_monitor — deterministic funnel health evaluation against charter thresholds #317-posted snapshot/charter/benchmarks fixtures ship in the bundle; test_e2e_matches_expected_findings_exactly asserts exact equality with the posted expected findings
  • Determinismtest_repeat_execution_is_bit_identical (serialized-output equality); no network modules imported (statically asserted)
  • Honesty floorinsufficient_data iff the declared floor is unmet, boundary-tested at the minimum denominator (25 computes, 24 refuses); declared metrics only, never inferred
  • Agent-loop contract — stated in instructions.md and on the catalog page (error → host blocks until operator override; warning → surface, never block; insufficient_data → treat as absent, never substitute)
  • kb/ benchmarks-as-datakb/benchmarks_demo.json (timestamped, sourced, versioned; synthetic demo values); revisions land as data-only PRs, proposer-maintained
  • Complete documentation surface — catalog page with version metadata, Skill history, and Usage Examples for all five providers; registry index row; agent-loops matrix row; extras sync

No runnable script is added under examples/, so examples/README.md is intentionally untouched (the in-bundle fixtures are the end-to-end example pack per #317).

External impact (quantified)

  • New (27 files): skills/monitoring/kpi_gate/ — 25 bundle files (__init__.py, manifest.yaml, skill.py, instructions.md, card.json, test_skill.py, 3 reference schemas, 1 kb demo data file, 15 fixtures) — plus docs/skills/kpi_gate.md and tests/fixtures/card_ui_schema/monitoring__kpi_gate.json
  • Existing files changed (4, one hunk each): CHANGELOG.md 1 entry under [Unreleased]; docs/skills/README.md 1 index row; docs/usage/agent_loops.md 1 matrix row; pyproject.toml 1 generated extras line (monitoring_kpi_gate = [], from scripts/sync_extras.py)
  • skillware/core/ and all other skills: untouched

Removal procedure

Deleting skills/monitoring/kpi_gate/, docs/skills/kpi_gate.md, and tests/fixtures/card_ui_schema/monitoring__kpi_gate.json, then reverting the four one-hunk edits listed above (CHANGELOG entry, README row, agent-loops row, generated extras line), removes this contribution completely; no other file depends on it.

Notes for review

Type of Change

  • New Skill — new registry bundle under skills/
  • Skill Upgrade — changes to an existing skill under skills/
  • Bug Fix — incorrect runtime or framework behavior
  • Documentation — docs, README, CONTRIBUTING only
  • Framework Featureskillware/core/ loader, env, adapters
  • CLIskillware/cli.py, docs/usage/cli.md
  • Examplesexamples/*.py, agent loops, examples/README.md
  • Packaging — PyPI wheel, pyproject.toml, MANIFEST.in
  • RFC / meta — templates, labels, CI, or large design doc

Checklist (all PRs)

  • Linked GitHub issue (Fixes #… or Refs #…)
  • Scope matches the issue — no unrelated refactors
  • python -m black --check . and flake8 pass locally (or CI-equivalent subset)
  • pytest skills/ and pytest tests/ pass locally when relevant (286 and 259 passed)
  • CHANGELOG.md updated under [Unreleased] when user-visible behavior changes
  • examples/README.md updated if this PR adds, renames, or removes a runnable script (no script added; file untouched)
  • Ran pytest tests/test_registry_docs.py when skills, examples index, or agent-loops matrix changed (9 passed)

New or updated skill

Bundle and metadata

  • Skill at skills/<category>/<skill_name>/ (from templates/python_skill/ or equivalent)
  • manifest.yaml: name (full ID), version, description, parameters, constitution, real issuer
  • Optional: short_description, issuer.github, issuer.org, requirements, env_vars (short_description, github, org: AO, requirements: []; no env_vars — fully offline)

Logic, cognition, tests

  • Deterministic skill.py (no ad-hoc LLM-generated execution paths)
  • instructions.md explains when and how to use the skill
  • card.json issuer matches manifest when present
  • test_skill.py covers execution and schema expectations
  • SkillLoader.load_skill("<category>/<skill_name>") succeeds (or deps documented)

Documentation and catalog

  • docs/skills/<skill_name>.md and row in docs/skills/README.md
  • Usage Examples for Gemini, Claude, OpenAI, DeepSeek, Ollama per skill usage template

Constitution and safety (skills only)

Evaluate-only: no data fetching, no side effects, no automated remediation, no network in execute(). Refuses rather than guesses (insufficient_data with reason codes; no default substitution). Every finding carries code, metric, threshold, and observed value. Honest limits (no data acquisition, causal inference, or threshold optimization) documented in instructions.md and the catalog page.

Related Issues

Fixes #317

…LS#317 interface.

The bundle evaluates a metrics snapshot against an operator-maintained policy charter and optional versioned benchmark data, returning error, warning, and insufficient_data findings with fail-closed contract errors from a closed registry. Validation follows the four-stage order frozen in issue ARPAHLS#317, uses stdlib checks only (requirements stay empty), and ships the three reference JSON Schemas, demo benchmark data, end-to-end and fail-closed fixtures, 28 offline bundle tests, and the card UI schema fixture. Refs ARPAHLS#317
…y indexes.

Adds the catalog page with version metadata, the closed error registry, usage examples for all five providers, and an initial skill history row; adds the index row in docs/skills/README.md, the reference-scripts matrix row in docs/usage/agent_loops.md, the CHANGELOG entry under Unreleased, and the generated monitoring_kpi_gate extras line from scripts/sync_extras.py. Refs ARPAHLS#317
@rosspeili

Copy link
Copy Markdown
Contributor

Thanks @mrmasa88, this is huge, detailed, yet solid work on the discussion on #317.

Before merge (please):

  1. Drop any Co-authored-by: AI trailers on commits (repo policy, human attribution only). Your last commit is with Claude.
  2. instructions.md tone, opening line is close to persona-style (You are using…). Please align with skill-context guidance: lead with registry ID + contract/limits (catalog page style), like other monitoring skills.
  3. Issue [New Skill]: monitoring/funnel_monitor — deterministic funnel health evaluation against charter thresholds #317 title/Skill ID, PR ships monitoring/kpi_gate (not funnel_monitor). Fine if that rename was agreed in the thread; we’ll update the issue field on merge, confirm that’s intentional.
  4. No examples/*_demo.py, I guess acceptable given in-bundle fixtures, optional later local-execute demo + smoke row.
  • __init__.py export of KpiGateSkill optional but nice for consistency. I would include both in this PR.

Also run black and flake8, it showed 4 files if not more would be affected on my machine.

Once CI is green LGTM to merge. Nice work as always <3

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[New Skill]: monitoring/funnel_monitor — deterministic funnel health evaluation against charter thresholds

2 participants