feat(llm): cost telemetry + per-user daily token budget [STORY-012]#17
Merged
feat(llm): cost telemetry + per-user daily token budget [STORY-012]#17
Conversation
Adds three new building blocks under @learnpro/llm: - pricing.ts — versioned MODEL_PRICING table (Opus/Sonnet/Haiku) + costFor() calculator. Append-only convention: bump PRICING_VERSION + add a new constant when prices change. Unknown models record cost=0 + known_model=false so analytics can flag operator-stale tables without breaking the runtime path. - budget.ts — UsageStore interface + InMemoryUsageStore (keyed by user_id + UTC date). DailyTokenBudget with assertWithinBudget() + decideModel() (downgrade ladder: premium=Opus → mid=Sonnet → cheap=Haiku, kicks in at the threshold, default 80%) + record(). limit=0 means unlimited (self-hosted default); explicit req.model always wins. - budget-gated-provider.ts — decorator that wraps any LLMProvider: pre-call assertWithinBudget (throws TokenBudgetExceededError), pre-call decideModel (may downgrade), post-call record. Embed passes through (no per-user attribution for embeddings yet). LLMTelemetryEventSchema gains cost_usd, pricing_version, optional session_id / cached_tokens / tool_used. AnthropicProvider stamps cost via costFor() in recordTelemetry; toolCall populates tool_used from the first invocation. CompleteRequestSchema gains optional session_id. DB-backed UsageStore + agent_calls Drizzle migration + API 429 mapping are split into STORY-060 to keep STORY-012 within its S estimate (interface + decorator, no migration). Interfaces are stable; STORY-060 just adds Drizzle impls behind them. 38 new tests across 3 files; 72 tests passing in @learnpro/llm. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
MODEL_PRICINGtable for Opus/Sonnet/Haiku stamped withPRICING_VERSION = "2026-04-26";costFor()returnscost_usd=0+known_model=falsefor unknown models so analytics can flag operator-stale tables without breaking the runtime path.UsageStoreinterface +InMemoryUsageStorekeyed by(user_id, UTC date);DailyTokenBudgetwithassertWithinBudget()(throwsTokenBudgetExceededErrorat limit) +decideModel()(downgrade ladder Opus → Sonnet → Haiku, kicks in at threshold = 0.8).BudgetGatedLLMProviderdecorator — wraps anyLLMProvider: pre-call assert + decideModel (may downgrade), post-call record. Embed passes through.limit=0means unlimited (self-hosted default); explicitreq.modelalways wins.LLMTelemetryEventSchemagainscost_usd,pricing_version, optionalsession_id/cached_tokens/tool_used.AnthropicProvidernow stamps cost viacostFor();toolCallpopulatestool_usedfrom the first invocation.CompleteRequestSchemagains optionalsession_id.agent_callsDrizzle migration + DB-backed sink + API 429 mapping moved to STORY-060 to keep STORY-012 at its S estimate. Interfaces (UsageStore,LLMTelemetrySink) are stable; STORY-060 just adds Drizzle impls behind them.Acceptance criteria
LLMTelemetryEvent(provider, model, role, user_id, session_id, task, input/output_tokens, cached_tokens, cost_usd, pricing_version, tool_used, latency_ms, ok, decided_at, prompt_version) —agent_callstable is in STORY-060.BudgetGatedLLMProvider) — applied at the provider layer so any caller goes through it.MODEL_TIERSladder.TokenBudgetExceededErrorcarries a human-friendly message; HTTP 429 mapping in STORY-060.PRICING_VERSION = "2026-04-26".Test plan
pnpm --filter @learnpro/llm test— 72 passed / 1 skipped (integration test, needs ANTHROPIC_API_KEY).pricing.test.ts(6),budget.test.ts(20),budget-gated-provider.test.ts(12).pnpm typecheck— green across all 12 packages.pnpm lint— green across all 8 lintable packages.pnpm test— green across the monorepo.LEARNPRO_DAILY_TOKEN_LIMIT=100+ a real Anthropic key, hit the playground twice and observe the friendly 429 on call chore(meta): lift no-code rule, fix grace-days, add PR workflow #2.🤖 Generated with Claude Code