Skip to content

skillopt: MetaStatus never advances past "proposed" — the meta-skill does dedup, not learning #339

Description

@kaghni

src/skillify/skillopt-meta.ts documents the meta-skill as the optimizer's cross-run memory that should let the optimizer learn which edits help:

// When the A/B gate lands, the recorded `status` (proposed → applied/reverted)
// closes the loop so the optimizer learns which kinds of edits actually help
// a given skill.

MetaStatus is declared as "proposed" | "applied" | "reverted" (line 15), but only "proposed" is ever writtenskillopt-meta.ts:82. Nothing in skillopt-worker.ts or skillopt-improve.ts transitions an entry.

Consequences:

  • priorEditSummaries() feeds the proposer every previously-tried edit without any signal about whether it helped, so the proposer can't prefer edit shapes that worked.
  • The meta-skill's only working function today is dedup (alreadyProposed / fingerprint matching), which is real and valuable — but it's not what the header claims.
  • The optimizer has online evaluation (every invocation is judged, including invocations of a version it just published) but no attribution: it never compares v_n vs v_n+1 on the same task, so it cannot say a given edit caused an improvement.

Options:

  1. Cheapest, no A/B needed: on the next judged invocation of skill X at version v+1, write back applied (judge passed) or reverted (judge failed again) against the meta entry that produced v+1. That's a real outcome signal from data already being collected — it's sequential, not controlled, but it's honest and free.
  2. Full A/B gate as originally designed (expensive — re-runs a user's task against two versions on their machine/token budget).
  3. If neither is planned soon, narrow the type to "proposed" and update the header so the code stops advertising a capability it doesn't have.

Option 1 seems worth doing regardless; it turns priorEditSummaries from a blocklist into an actual prior.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions