src/skillify/skillopt-meta.ts documents the meta-skill as the optimizer's cross-run memory that should let the optimizer learn which edits help:
// When the A/B gate lands, the recorded `status` (proposed → applied/reverted)
// closes the loop so the optimizer learns which kinds of edits actually help
// a given skill.
MetaStatus is declared as "proposed" | "applied" | "reverted" (line 15), but only "proposed" is ever written — skillopt-meta.ts:82. Nothing in skillopt-worker.ts or skillopt-improve.ts transitions an entry.
Consequences:
priorEditSummaries() feeds the proposer every previously-tried edit without any signal about whether it helped, so the proposer can't prefer edit shapes that worked.
- The meta-skill's only working function today is dedup (
alreadyProposed / fingerprint matching), which is real and valuable — but it's not what the header claims.
- The optimizer has online evaluation (every invocation is judged, including invocations of a version it just published) but no attribution: it never compares v_n vs v_n+1 on the same task, so it cannot say a given edit caused an improvement.
Options:
- Cheapest, no A/B needed: on the next judged invocation of skill X at version v+1, write back
applied (judge passed) or reverted (judge failed again) against the meta entry that produced v+1. That's a real outcome signal from data already being collected — it's sequential, not controlled, but it's honest and free.
- Full A/B gate as originally designed (expensive — re-runs a user's task against two versions on their machine/token budget).
- If neither is planned soon, narrow the type to
"proposed" and update the header so the code stops advertising a capability it doesn't have.
Option 1 seems worth doing regardless; it turns priorEditSummaries from a blocklist into an actual prior.
src/skillify/skillopt-meta.tsdocuments the meta-skill as the optimizer's cross-run memory that should let the optimizer learn which edits help:MetaStatusis declared as"proposed" | "applied" | "reverted"(line 15), but only"proposed"is ever written —skillopt-meta.ts:82. Nothing inskillopt-worker.tsorskillopt-improve.tstransitions an entry.Consequences:
priorEditSummaries()feeds the proposer every previously-tried edit without any signal about whether it helped, so the proposer can't prefer edit shapes that worked.alreadyProposed/ fingerprint matching), which is real and valuable — but it's not what the header claims.Options:
applied(judge passed) orreverted(judge failed again) against the meta entry that produced v+1. That's a real outcome signal from data already being collected — it's sequential, not controlled, but it's honest and free."proposed"and update the header so the code stops advertising a capability it doesn't have.Option 1 seems worth doing regardless; it turns
priorEditSummariesfrom a blocklist into an actual prior.