Add the monitoring/kpi_gate skill bundle implementing the issue #317 interface - #318
Open
mrmasa88 wants to merge 2 commits into
Open
Add the monitoring/kpi_gate skill bundle implementing the issue #317 interface#318mrmasa88 wants to merge 2 commits into
mrmasa88 wants to merge 2 commits into
Conversation
…LS#317 interface. The bundle evaluates a metrics snapshot against an operator-maintained policy charter and optional versioned benchmark data, returning error, warning, and insufficient_data findings with fail-closed contract errors from a closed registry. Validation follows the four-stage order frozen in issue ARPAHLS#317, uses stdlib checks only (requirements stay empty), and ships the three reference JSON Schemas, demo benchmark data, end-to-end and fail-closed fixtures, 28 offline bundle tests, and the card UI schema fixture. Refs ARPAHLS#317
…y indexes. Adds the catalog page with version metadata, the closed error registry, usage examples for all five providers, and an initial skill history row; adds the index row in docs/skills/README.md, the reference-scripts matrix row in docs/usage/agent_loops.md, the CHANGELOG entry under Unreleased, and the generated monitoring_kpi_gate extras line from scripts/sync_extras.py. Refs ARPAHLS#317
Contributor
|
Thanks @mrmasa88, this is huge, detailed, yet solid work on the discussion on #317. Before merge (please):
Also run black and flake8, it showed 4 files if not more would be affected on my machine. Once CI is green LGTM to merge. Nice work as always <3 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds the
monitoring/kpi_gateregistry skill implementing the interface agreed in #317 (three input JSON Schemas, output contract, four-stage fail-closed validation order, agent-loop contract, and the end-to-end example). Design rationale and the frozen interface live in the #317 comments and are not re-argued here.Acceptance criteria mapped to the diff:
error/warning/insufficient_datawith reason codes) —skills/monitoring/kpi_gate/skill.py; branch coverage intest_skill.py(28 offline tests)schemas/as reference documents;requirements: []stays empty by design, so no runtimejsonschemadependency); closed error registryINVALID_METRICS_SCHEMA,INVALID_POLICY_SCHEMA,INVALID_BENCHMARKS_SCHEMA,NO_METRICS_PROVIDED,UNKNOWN_METRIC_KEY,UNKNOWN_RULE_METRIC,UNKNOWN_DENOMINATOR_METRIC,BENCHMARK_VERSION_MISSING,BENCHMARK_REF_UNRESOLVED— one dedicated test per code, including thedependentRequiredrejectiontest_e2e_matches_expected_findings_exactlyasserts exact equality with the posted expected findingstest_repeat_execution_is_bit_identical(serialized-output equality); no network modules imported (statically asserted)insufficient_dataiff the declared floor is unmet, boundary-tested at the minimum denominator (25 computes, 24 refuses); declared metrics only, never inferredinstructions.mdand on the catalog page (error→ host blocks until operator override;warning→ surface, never block;insufficient_data→ treat as absent, never substitute)kb/benchmarks_demo.json(timestamped, sourced, versioned; synthetic demo values); revisions land as data-only PRs, proposer-maintainedNo runnable script is added under
examples/, soexamples/README.mdis intentionally untouched (the in-bundle fixtures are the end-to-end example pack per #317).External impact (quantified)
skills/monitoring/kpi_gate/— 25 bundle files (__init__.py,manifest.yaml,skill.py,instructions.md,card.json,test_skill.py, 3 reference schemas, 1 kb demo data file, 15 fixtures) — plusdocs/skills/kpi_gate.mdandtests/fixtures/card_ui_schema/monitoring__kpi_gate.jsonCHANGELOG.md1 entry under[Unreleased];docs/skills/README.md1 index row;docs/usage/agent_loops.md1 matrix row;pyproject.toml1 generated extras line (monitoring_kpi_gate = [], fromscripts/sync_extras.py)skillware/core/and all other skills: untouchedRemoval procedure
Deleting
skills/monitoring/kpi_gate/,docs/skills/kpi_gate.md, andtests/fixtures/card_ui_schema/monitoring__kpi_gate.json, then reverting the four one-hunk edits listed above (CHANGELOG entry, README row, agent-loops row, generated extras line), removes this contribution completely; no other file depends on it.Notes for review
monitoring/kpi_gatefollows the "business-KPI gate" framing from the [New Skill]: monitoring/funnel_monitor — deterministic funnel health evaluation against charter thresholds #317 thread; the name is confined to grep-replaceable locations (paths,manifest.name, docs rows, fixture filename, extras line, schema$ids) if a different candidate is preferred.*(pending merge)*for the commit SHA, following thedeceptive_ui_guardprecedent.detailkeeps the exact [New Skill]: monitoring/funnel_monitor — deterministic funnel health evaluation against charter thresholds #317 shape for the canonical below-minimum-denominator refusal; shapes [New Skill]: monitoring/funnel_monitor — deterministic funnel health evaluation against charter thresholds #317 did not pin (granularity floor, missing denominator) additionally carrydetail.unmet_floornaming the refused floor — documented ininstructions.mdand the catalog page, tested in all four shapes.check.applies_tois shape-validated and echoed but reserved (no evaluation semantics in v1), stated explicitly ininstructions.mdand the catalog page.Type of Change
skills/skills/skillware/core/loader, env, adaptersskillware/cli.py,docs/usage/cli.mdexamples/*.py, agent loops,examples/README.mdpyproject.toml,MANIFEST.inChecklist (all PRs)
Fixes #…orRefs #…)python -m black --check .andflake8pass locally (or CI-equivalent subset)pytest skills/andpytest tests/pass locally when relevant (286 and 259 passed)CHANGELOG.mdupdated under[Unreleased]when user-visible behavior changesexamples/README.mdupdated if this PR adds, renames, or removes a runnable script (no script added; file untouched)pytest tests/test_registry_docs.pywhen skills, examples index, or agent-loops matrix changed (9 passed)New or updated skill
Bundle and metadata
skills/<category>/<skill_name>/(fromtemplates/python_skill/or equivalent)manifest.yaml:name(full ID),version,description,parameters,constitution, realissuershort_description,issuer.github,issuer.org,requirements,env_vars(short_description,github,org: AO,requirements: []; noenv_vars— fully offline)Logic, cognition, tests
skill.py(no ad-hoc LLM-generated execution paths)instructions.mdexplains when and how to use the skillcard.jsonissuer matches manifest when presenttest_skill.pycovers execution and schema expectationsSkillLoader.load_skill("<category>/<skill_name>")succeeds (or deps documented)Documentation and catalog
docs/skills/<skill_name>.mdand row indocs/skills/README.mdConstitution and safety (skills only)
Evaluate-only: no data fetching, no side effects, no automated remediation, no network in
execute(). Refuses rather than guesses (insufficient_datawith reason codes; no default substitution). Every finding carries code, metric, threshold, and observed value. Honest limits (no data acquisition, causal inference, or threshold optimization) documented ininstructions.mdand the catalog page.Related Issues
Fixes #317