[New Skill]: monitoring/funnel_monitor — deterministic funnel health evaluation against charter thresholds

[mrmasa88]

Skill ID

monitoring/funnel_monitor

Category

monitoring

What should this skill do?

Summary

• A deterministic skill that evaluates a weekly funnel metrics snapshot — outreach sends, replies, bookings, landing-page sessions, downloads (all weekly counts), per-prospect stage states, and content-asset publication dates (for freshness rules) — against the thresholds an operator's business charter defines and against industry-doctrine benchmark values, returning findings. It applies the threshold rules derived from the charter to the snapshot.

• The finding vocabulary has two severities and one third state:

• Charter violation = error (blocking) — a breach of a constraint the operator set for themselves

• Deviation from industry doctrine = warning (visible, non-blocking) — notice of drift from a benchmark

• What cannot be computed honestly is not computed = insufficient_data (a refusal with a reason code) — no vanity ratios from immature data (e.g. "15% conversion" at n=20). The most opinionated behavior in this skill.

• execute() is a pure function and performs no network calls. Identical input always returns identical output.

Motivation / use cases

• A monitoring capability AO's agents will use in production. It is the first skill manufactured by AO's distillation pipeline (Skill Foundry) from requirement extraction onward, proposed as an AO-originated distillation.

• Where the existing monitoring/token_limiter covers resource-side monitoring, this adds business-metric monitoring to the category.

• Aimed at solo operators and small teams (coaches, course creators, consultants, e-commerce operators). When one person does the outreach, the production, and the delivery, that same person is also the only one watching the numbers — which means, in practice, nobody is. This makes that watching a machine's job.

• First production deployment planned on a real company: the funnel and price ladder of OrbLabs (a due-diligence firm), run on a cadence. The operational ledger will be accumulated in a publishable form as the evidence base for future versions.

Ideal Inputs & Outputs

Proposed behavior

• Input: weekly funnel metrics snapshot (JSON) + a charter parameter file (versioned YAML)
• Parameter resolution: deterministic binding via a self-describing schema — each parameter carries applies_to / binds_to / schema_version, with support for an explicit unlimited literal. Shipping semantics alongside values prevents underdetermination at extraction and validation time.
• Benchmarks: industry benchmark values are separated from code as versioned data files (benchmarks-as-data), placed under the bundle's kb/ with timestamps and sources, following the existing kb dataset conventions. Value updates land as ordinary data-revision PRs, versioned independently of logic changes.
• Output: status envelope + findings[]. Each finding carries:
  • finding — an UPPER_SNAKE_CASE code from a closed set enumerated in the bundle (e.g. NO_BOOKING)
  • severity: error | warning, or state: insufficient_data
  • detail — the evaluated metric, threshold, and observed value
  • action — for error: blocked — requires operator override; for warning: none — surfaced for the operator
  • reason — on refusal, a reason code (e.g. required_granularity_unavailable, cohort_attribution_unavailable)
• Fail-closed: runtime contract violations (unknown enums, missing fields, type mismatches, benchmark-file inconsistencies) fail with codes from a closed error registry (e.g. FUNNEL_MODE_REQUIRED, UNKNOWN_METRIC_KEY, NO_METRICS_PROVIDED, BENCHMARK_VERSION_MISSING). No default substitution, no improvised messages. Contract violations (errors) and honest non-computability (insufficient_data) are returned as distinct things.

Parameters (example)

schema_version: 2
floor_price:
  amount: 6000
  applies_to: [one_to_one]     # the value declares its own scope — this floor
                               # governs the 1:1 tier only, and does not reach
                               # the $349 self-paced product
capacity_limits:
  max_one_to_one_clients:
    value: 6
    binds_to: sum_of_one_to_one_capacities   # the value declares what it counts
tier_structure:
  - {name: Foundations of Everyday Leadership, price_usd: 349, capacity: unlimited}   # explicit literal; 0 is never a sentinel

Target runtime

Model agnostic (all supported adapters)

External APIs & env vars (if any)

v1 ship bar

• The full three-state vocabulary (error / warning / insufficient_data with reason codes) implemented

• Charter parameter file (self-describing schema) + kb/ benchmark data (timestamped, sourced)

• Fully mocked bundle tests, covering fail-closed paths and per-reason-code fixtures

• Complete documentation surface: catalog page (Version / Skill history / usage examples for all five providers) and extras sync

Constitution

• Evaluate only — no data fetching, no side effects, no automated remediation

• Deterministic, offline core — no network in execute(); identical input → identical output

• Refuse rather than guess — what cannot be computed honestly returns insufficient_data with a reason code; no default substitution

• Every finding explained — code, metric, threshold, and observed value always attached

• Honest limits documented — what this skill does not do (data acquisition, causal inference, threshold optimization) is stated explicitly

Composition (not bundled)

• monitoring/token_limiter — resource-side and business-metric monitoring are complementary; no runtime coupling

• A pricing-validation sibling (separate proposal, planned) — shares the same charter parameter format; will be proposed in its own issue

• This skill runs standalone and never calls other skills internally; chaining guidance stays in the docs

v2 (planned)

• Benchmark data revisions driven by the OrbLabs operational ledger — threshold and false-positive tuning as data-only PRs

• Broader funnel-mode / metric coverage, added where operation demonstrates demand

Testing

• All tests fully mocked (CI never downloads)

• Benchmark data shipped as fixtures

• Fail-closed paths (unknown enum, missing field, type mismatch) explicitly covered

• Full branch coverage of the two severities plus the third state (including per-reason-code insufficient_data fixtures)

Issuer

• name: Masa / github: mrmasa88 / org: AO

[rosspeili]

Thanks for the detailed proposal @mrmasa88, the pattern looks good and from wht I can understand you propose a deterministic evaluator, structured findings, fail-closed validation, insufficient_data instead of vanity metrics. I think it would sit nicely next to monitoring/token_limiter as a business-KPI gate vs a token budget gate.

Before approval for public registry scope, we need a clearer split between a reusable generic skill and your OrbLabs / coaching-funnel vertical.

What I understand: something upstream builds a weekly metrics snapshot (JSON), then the operator maintains a charter (YAML) with their thresholds, the skill compares snapshot vs charter + optional kb/ benchmarks and returns findings[] (error / warning / insufficient_data). No network, pure python functions.

Important clarification: the skill is an auditor in this case, not a data pipeline, in the sense it does not fetch CRM/email/landing-page data, etc. right? I assume the snapshot must come from outside, say exports, scripts, dashboards, or an agent that already assembled counts. The charter is typically operator-maintained config (probably edited manually, versioned in git, passed into execute()). It also does not magically accept arbitrary JSON/YAML, and v1 needs strict schemas, a closed rule set, and fail-closed errors when inputs don’t match, plus insufficient_data when a metric can’t be computed honestly as you propose.

I see two paths:

1. Public registry, ship a generic skill (eg. monitoring/policy_evaluator or monitoring/metrics_threshold_gate, or monitoring/KPI_policy_gate or something), with the core being { metrics, policy, benchmarks? } → findings[]. Your coaching funnel / OrbLabs charter becomes an example pack (fixtures + docs), not the skill identity. Others bring their own snapshot + charter, you customize via config, not forks.

2. Internal / private, if OrbLabs is the only near-term consumer, build funnel_monitor in AO’s private skill root and skip the public registry until a second independent use case exists or unless we take path 1, which will be fixed for generic use. 2 also valid nevertheless if you wanna build this for AO specifically.

For Path A we'd need: one full end-to-end example (snapshot JSON + charter YAML + expected findings), generic naming (not funnel-specific), small v1 rule set + mocked tests, explicit agent-loop contract (what the host does on each severity), and clarity on who maintains benchmark data in kb/.

Tips: defer pricing sibling + ledger-driven v2 for now, consider one policy framework if the charter format is shared, spike/RFC with JSON Schema before a full bundle PR if you wanna discuss this or think around it first.

Happy to help and revisit once the interface is concrete. <3

[mrmasa88]

Thanks Ross! @rosspeili — Path A, no hesitation. The generic core is what Skill Foundry
produces anyway; the funnel charter was only ever the first payload.

Concretely:

• Name: monitoring/kpi_gate (taking it straight from your "business-KPI
  gate" framing — happy with any of your three candidates if you prefer).
• Core contract: { metrics, policy, benchmarks? } → findings[], strict
  JSON Schema on all three inputs, closed rule set, fail-closed errors,
  insufficient_data with reason codes.
• Agent-loop contract (explicit): error → host blocks the dependent
  action until operator override; warning → surface to operator, never
  block; insufficient_data → treat the metric as absent — the host must not
  substitute or estimate.
• Coaching-funnel / OrbLabs charter → example pack: fixtures + docs only,
  clearly labeled synthetic. Skill identity stays generic.
• kb/ maintenance: I maintain benchmark data via ordinary data-revision
  PRs (timestamped, sourced) — that is exactly what benchmarks-as-data was
  designed for. If a second party later wants to co-maintain, the format
  supports it.
• Deferred per your tips: pricing sibling and ledger-driven v2 out of
  this issue.

Next: I'll post the JSON Schemas for the three inputs plus one full E2E
example (snapshot + charter + expected findings) as a comment here before
opening the bundle PR.

[mrmasa88]

@rosspeili Hey!

As promised — the concrete interface: JSON Schemas for the three inputs, the output contract, and one full end-to-end example. Name used below: monitoring/kpi_gate (pending your ack — happy with any of your three candidates).

Design split that keeps the skill generic

• The skill's own closed error registry covers contract violations only (UNKNOWN_METRIC_KEY, NO_METRICS_PROVIDED, BENCHMARK_VERSION_MISSING, …). These are the skill's identity and never change per operator.
• Finding codes come from the policy file: each charter declares its own closed set of UPPER_SNAKE_CASE rule ids, validated against a strict pattern at load. So NO_BOOKING is charter content, not skill code — another operator's charter declares NO_SIGNED_CONTRACT instead, and the skill needs no fork.
• Benchmarks are optional. A charter that references no benchmark runs fully self-contained.

1. metrics — snapshot input (JSON)

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "skillware:monitoring/kpi_gate:metrics:1",
  "type": "object",
  "required": ["schema_version", "period", "metrics"],
  "additionalProperties": false,
  "properties": {
    "schema_version": {"const": 1},
    "period": {
      "type": "object",
      "required": ["start", "end", "granularity"],
      "additionalProperties": false,
      "properties": {
        "start": {"type": "string", "format": "date"},
        "end": {"type": "string", "format": "date"},
        "granularity": {"enum": ["daily", "weekly", "monthly"]}
      }
    },
    "metrics": {
      "type": "object",
      "minProperties": 1,
      "patternProperties": {"^[a-z][a-z0-9_]*$": {"type": "number", "minimum": 0}},
      "additionalProperties": false
    }
  }
}

Notes: every key in metrics must be declared by the policy (unknown → UNKNOWN_METRIC_KEY, fail-closed). Richer inputs (per-entity stage states, asset freshness) are representable in v1 as precomputed metrics; structured entity input is a possible v1.1 extension, out of scope now.

2. policy — charter input (YAML, shown as schema)

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "skillware:monitoring/kpi_gate:policy:2",
  "type": "object",
  "required": ["schema_version", "policy_id", "metrics", "rules"],
  "additionalProperties": false,
  "properties": {
    "schema_version": {"const": 2},
    "policy_id": {"type": "string"},
    "metrics": {
      "description": "Closed declaration of every metric key this policy may reference.",
      "type": "array", "minItems": 1,
      "items": {"type": "string", "pattern": "^[a-z][a-z0-9_]*$"}
    },
    "rules": {
      "type": "array", "minItems": 1,
      "items": {
        "type": "object",
        "required": ["id", "metric", "check", "severity"],
        "additionalProperties": false,
        "properties": {
          "id": {"type": "string", "pattern": "^[A-Z][A-Z0-9_]*$"},
          "metric": {"type": "string"},
          "check": {
            "oneOf": [
              {"type": "object", "required": ["op", "threshold"], "additionalProperties": false,
               "properties": {"op": {"enum": ["gte", "lte", "gt", "lt", "eq"]},
                              "threshold": {"type": ["number", "string"]},
                              "applies_to": {"type": "array", "items": {"type": "string"}}}},
              {"type": "object", "required": ["op", "benchmark_ref"], "additionalProperties": false,
               "properties": {"op": {"enum": ["gte", "lte"]},
                              "benchmark_ref": {"type": "string"},
                              "tolerance_pct": {"type": "number", "minimum": 0}}}
            ]
          },
          "severity": {"enum": ["error", "warning"]},
          "requires": {
            "description": "Honesty floor: unmet ⇒ insufficient_data, never a guess. Floors are checked against the DECLARED denominator metric only — the evaluator never infers which metric backs a rate.",
            "type": "object", "additionalProperties": false,
            "properties": {
              "denominator_metric": {"type": "string", "pattern": "^[a-z][a-z0-9_]*$"},
              "min_denominator": {"type": "number", "minimum": 1},
              "granularity": {"enum": ["daily", "weekly", "monthly"]},
              "reason_code": {"type": "string", "pattern": "^[a-z][a-z0-9_]*$"}
            },
            "dependentRequired": {"min_denominator": ["denominator_metric"]}
          }
        }
      }
    }
  }
}

Notes: threshold accepts the explicit literal "unlimited" (never 0 as a sentinel). Derived-ratio metrics (e.g. a rate) are declared like any other metric and computed upstream or via a declared ratio_of: [num, den] extension — kept out of v1 core to keep the rule set small; the honesty floor (requires.min_denominator) is what gates them either way.

3. benchmarks — optional kb/ data (versioned)

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "skillware:monitoring/kpi_gate:benchmarks:1",
  "type": "object",
  "required": ["schema_version", "benchmark_version", "as_of", "sources", "values"],
  "additionalProperties": false,
  "properties": {
    "schema_version": {"const": 1},
    "benchmark_version": {"type": "string"},
    "as_of": {"type": "string", "format": "date"},
    "sources": {"type": "array", "minItems": 1, "items": {"type": "string", "format": "uri"}},
    "values": {
      "type": "object", "minProperties": 1,
      "patternProperties": {
        "^[a-z][a-z0-9_]*$": {
          "type": "object", "required": ["value"], "additionalProperties": false,
          "properties": {"value": {"type": "number"},
                         "unit": {"type": "string"},
                         "denominator_scope": {"type": "string"}}
        }
      },
      "additionalProperties": false
    }
  }
}

Ships under the bundle's kb/ with timestamps + sources; revisions land as ordinary data-only PRs. I maintain it (proposer-maintained); the format supports co-maintainers if a second party arrives.

4. Output contract

{"status": "completed", "policy_id": "…", "benchmark_version": "…|null",
 "findings": [
   {"finding": "<RULE_ID from the charter's closed set>",
    "severity": "error|warning",
    "detail": {"metric": "…", "threshold": "…", "observed": 0},
    "action": "blocked — requires operator override | none — surfaced for the operator"},
   {"state": "insufficient_data",
    "detail": {"metric": "…"},
    "reason": "<reason_code declared in the rule's honesty floor>"}
 ]}

Contract violations return {"status": "error", "error": {"code": "<skill registry code>", "detail": "…"}} instead — errors and honest non-computability never mix.

Agent-loop contract (restating for the record): error → host blocks the dependent action until operator override; warning → surface, never block; insufficient_data → treat the metric as absent; the host must not substitute or estimate.

5. End-to-end example (synthetic coaching-funnel pack — the vertical as fixtures, not identity)

Snapshot (fixtures/example_snapshot.json):

{"schema_version": 1,
 "period": {"start": "2026-08-17", "end": "2026-08-23", "granularity": "weekly"},
 "metrics": {"outreach_sends": 40, "replies": 6, "bookings": 0,
             "lp_sessions": 180, "downloads": 22, "download_rate_pct": 12.2,
             "reply_to_booking_rate_pct": 0}}

Charter (fixtures/example_charter.yaml):

schema_version: 2
policy_id: coaching_funnel_demo_v1
metrics: [outreach_sends, replies, bookings, lp_sessions, downloads,
          download_rate_pct, reply_to_booking_rate_pct]
rules:
  - id: NO_BOOKING
    metric: bookings
    check: {op: gte, threshold: 1}
    severity: error
  - id: DOWNLOAD_RATE_BELOW_DOCTRINE
    metric: download_rate_pct
    check: {op: gte, benchmark_ref: lead_magnet_download_rate_pct}
    severity: warning
  - id: BOOKING_CONVERSION_TRACKED
    metric: reply_to_booking_rate_pct
    check: {op: gte, threshold: 10}
    severity: warning
    requires: {denominator_metric: replies, min_denominator: 25,
               granularity: weekly, reason_code: cohort_attribution_unavailable}

Benchmarks (kb/benchmarks_demo.json): lead_magnet_download_rate_pct: {value: 25.0, unit: "pct", denominator_scope: "unique landing-page sessions"}, as_of: 2026-08-01, one source URL.

Expected findings:

{"status": "completed", "policy_id": "coaching_funnel_demo_v1", "benchmark_version": "demo-1",
 "findings": [
   {"finding": "NO_BOOKING", "severity": "error",
    "detail": {"metric": "bookings", "threshold": ">= 1", "observed": 0},
    "action": "blocked — requires operator override"},
   {"finding": "DOWNLOAD_RATE_BELOW_DOCTRINE", "severity": "warning",
    "detail": {"metric": "download_rate_pct", "threshold": ">= 25.0 (benchmark)", "observed": 12.2},
    "action": "none — surfaced for the operator"},
   {"state": "insufficient_data",
    "detail": {"metric": "reply_to_booking_rate_pct"},
    "reason": "cohort_attribution_unavailable"}
 ]}

(All values synthetic. The full pack — these fixtures plus docs — ships as the example, per Path A.)

Validation order (fail-closed, deterministic)

1. Schema-validate all three inputs → contract error on first failure
2. Cross-checks: every rule metric ∈ policy metrics; every requires.denominator_metric ∈ policy metrics; every snapshot key ∈ policy metrics (else UNKNOWN_METRIC_KEY); every benchmark_ref resolves (else BENCHMARK_VERSION_MISSING)
3. Honesty floors → insufficient_data iff snapshot[denominator_metric] < min_denominator, with the rule's declared reason code — checked against the declared metric only, never inferred
4. Rule evaluation in declared order → findings; identical input, identical output

If this shape works for you, I'll open the bundle PR against this interface (strict schemas in-bundle, mocked tests over these exact fixtures).