Run skill-creator eval loop

[jayendra13]

Context

The skill is published and working. Next step is running the full skill-creator eval loop to measure quality with-skill vs without-skill, then iterate based on feedback.

Prerequisites

• Working CECIL_API_KEY in .env (current key returns 401)
• skill-creator plugin installed at ~/.claude/plugins/marketplaces/claude-plugins-official/plugins/skill-creator/
• Python deps installed: uv add cecil xarray-sql matplotlib pandas

Where to put eval outputs

Per the skill-creator convention, the eval workspace is a sibling directory to the skill — not inside it:

your-project/
├── .claude/skills/cecil-data-analysis/    # the skill (cloned repo)
└── cecil-data-analysis-workspace/         # eval outputs (gitignored)
    └── iteration-1/
        ├── eval-1-lulc-year-over-year/
        │   ├── eval_metadata.json
        │   ├── with_skill/
        │   │   ├── run-1/
        │   │   │   ├── outputs/           # result.csv, result.md, result.png
        │   │   │   ├── grading.json
        │   │   │   └── timing.json
        │   │   ├── run-2/
        │   │   └── run-3/
        │   └── without_skill/
        │       ├── run-1/
        │       ├── run-2/
        │       └── run-3/
        ├── eval-2-biomass-trend-borneo/
        │   └── (same structure)
        ├── eval-3-iucn-species-overlap/
        ├── eval-4-soil-moisture-iowa/
        ├── eval-5-out-of-scope-weather/
        ├── benchmark.json
        ├── benchmark.md
        └── feedback.json

The skill's .gitignore already excludes *-workspace/.

JSON schemas for eval files

evals/evals.json (already exists in the repo)

{
  "skill_name": "cecil-data-analysis",
  "evals": [
    {
      "id": 1,
      "prompt": "User's task prompt",
      "expected_output": "Description of expected result",
      "files": [],
      "expectations": ["The output includes X", "The skill used script Y"]
    }
  ]
}

eval_metadata.json (one per eval directory)

{
  "eval_id": 1,
  "eval_name": "lulc-year-over-year",
  "prompt": "For my project AOI, what was the dominant land cover class in 2023...",
  "assertions": [
    "Result mentions a specific land cover class name, not just an integer code",
    "SQL JOINs against a ref_* table"
  ]
}

grading.json (one per run — field names must be exact)

{
  "expectations": [
    {"text": "...", "passed": true, "evidence": "Found in output: ..."},
    {"text": "...", "passed": false, "evidence": "Not found in any output"}
  ],
  "summary": {
    "passed": 2,
    "failed": 1,
    "total": 3,
    "pass_rate": 0.67
  }
}

Important: use text/passed/evidence — not name/met/details. The viewer depends on these exact keys.

timing.json (captured from subagent completion notification)

{
  "total_tokens": 84852,
  "duration_ms": 23332,
  "total_duration_seconds": 23.3
}

benchmark.json (generated by aggregate script)

{
  "metadata": {
    "skill_name": "cecil-data-analysis",
    "timestamp": "...",
    "runs_per_configuration": 3
  },
  "run_summary": {
    "with_skill": {
      "pass_rate": {"mean": 0.85, "stddev": 0.05},
      "time_seconds": {"mean": 45.0, "stddev": 12.0},
      "tokens": {"mean": 3800, "stddev": 400}
    },
    "without_skill": {
      "pass_rate": {"mean": 0.35, "stddev": 0.08},
      "time_seconds": {"mean": 32.0, "stddev": 8.0},
      "tokens": {"mean": 2100, "stddev": 300}
    },
    "delta": {
      "pass_rate": "+0.50",
      "time_seconds": "+13.0",
      "tokens": "+1700"
    }
  }
}

Steps

1. Spawn all runs in parallel

For each of the 5 prompts in evals/evals.json, spawn two subagents in the same turn — one with the skill, one without:

Eval	Prompt	Type
eval-1-lulc-year-over-year	dominant land cover class 2023 vs 2020	positive
eval-2-biomass-trend-borneo	aboveground biomass trend since 2018	positive
eval-3-iucn-species-overlap	IUCN red-listed species ranges (vector dataset)	positive
eval-4-soil-moisture-iowa	soil moisture April-May comparison	positive
eval-5-out-of-scope-weather	Berlin weather forecast	negative control

2. Write eval_metadata.json per test case

3. Draft assertions while runs are in progress

eval-1: result uses class names not integer codes, SQL JOINs ref table, covers both years, includes interpretation
eval-2: picks a biomass dataset, time series from 2018+, chart generated, trend direction described
eval-3: uses load_dataframe (vector), stops at subscription gate if no sub exists
eval-4: picks soil moisture dataset, filters Apr-May for two years, comparison + summary
eval-5: politely declines, no create_subscription called, no fabricated dataset IDs

4. Capture timing from each subagent notification

Save timing.json immediately when each subagent completes — this is the only chance to capture it.

5. Grade

Use agents/grader.md from skill-creator. Save grading.json per run.

6. Aggregate

cd ~/.claude/plugins/marketplaces/claude-plugins-official/plugins/skill-creator/skills/skill-creator
python -m scripts.aggregate_benchmark <workspace>/iteration-1 --skill-name cecil-data-analysis

7. Analyst pass

Look for non-discriminating assertions, high-variance evals, time/token tradeoffs. See agents/analyzer.md.

8. Launch eval viewer

python ~/.claude/plugins/marketplaces/claude-plugins-official/plugins/skill-creator/skills/skill-creator/eval-viewer/generate_review.py \
  <workspace>/iteration-1 \
  --skill-name cecil-data-analysis \
  --benchmark <workspace>/iteration-1/benchmark.json

For iteration 2+, add --previous-workspace <workspace>/iteration-<N-1>.

9. Read feedback and iterate

10. Optional: run description optimization via scripts.run_loop

Reference documentation

All official docs live in the skill-creator plugin:

File	What it covers
skill-creator/SKILL.md	Full eval workflow (spawn, grade, aggregate, iterate)
skill-creator/references/schemas.md	Exact JSON schemas for all eval files
skill-creator/agents/grader.md	How to evaluate assertions against outputs
skill-creator/agents/analyzer.md	How to analyze benchmark results for patterns
skill-creator/agents/comparator.md	Blind A/B comparison (optional, advanced)
skill-creator/scripts/aggregate_benchmark.py	Aggregates runs into benchmark.json
skill-creator/eval-viewer/generate_review.py	Interactive HTML review viewer

All paths relative to ~/.claude/plugins/marketplaces/claude-plugins-official/plugins/skill-creator/skills/.

Notes

• Eval loop will make real Cecil API calls. Eval-3 (IUCN species) may trigger subscription creation — confirm before running.
• Eval-5 is a negative control — pass = skill declines.
• Full step-by-step details in evals/RUN.md.