[FEATURE] Unified LLM Observability & Interfaces — Cost Tracking, Streaming, Evaluation, and Natural Language Querying

[dorismaduegbunam]

Closes #452
Closes #450
Closes #448
Closes #446

This PR implements a comprehensive suite of features to enhance LLM capabilities, observability, and cost control across the AstroML platform:

1. Cost Tracking & Budgeting ([FEATURE] LLM Cost Tracking & Budgeting — Monitor and optimize API spend #452): Real-time spend tracking, tier-based limits (Free, Pro, Enterprise), model permission rules, alerts (50%, 80%, 100%), and dynamic cost-saving model routing.
2. Streaming Response Handler ([FEATURE] Streaming Response Handler — Real-time streaming for LLM outputs #450): Standardized async stream buffering with backpressure control, Server-Sent Events (SSE) for REST, bidirectional WebSockets, and generation latency tracking.
3. LLM Evaluation Framework ([FEATURE] LLM Evaluation Framework — Benchmark and score LLM outputs for quality #448): Automated metric suite (unigram BLEU, ROUGE-L, factuality, relevance, coherence, safety), test dataset runners, human review feedback workflows, regression detectors, and markdown reporting.
4. Natural Language Query Interface ([FEATURE] Natural Language Query Interface — SQL and pipeline queries via natural language #446): Read-only schema-aware NL-to-SQL translation, ML experiment pipeline YAML config generation, strict complexity & timeout validators, and context-based suggestions.

Key Changes

Observability & Infrastructure

• cost.py (astroml/db/models): SQLAlchemy models for logging request costs (LLMCostRecord) and tracking limits (LLMBudget).
• tracker.py (astroml/llm/cost): Tracks real-time token counts and pricing rates for providers.
• budget.py & optimizer.py (astroml/llm/cost): Budget rules, tier restrictions, and low-complexity/high-spend model routing optimization.
• alerts.py & analytics.py (astroml/llm/cost): Threshold alerts (50%, 80%, 100%), historical reports, and spending forecasts.

Streaming Response

• buffer.py & handler.py (astroml/llm/streaming): Cancellation-safe stream processing and backpressure handling.
• formatter.py & aggregator.py (astroml/llm/streaming): Multi-stream merging, SSE REST endpoints, and WebSocket payload generators.

Evaluation & Quality

• metrics.py & framework.py (astroml/llm/eval): Core evaluation orchestration covering BLEU, ROUGE-L, factuality, coherence, safety, and judge-scoring.
• regression.py & human.py (astroml/llm/eval): Quality regression detection for CI/CD and annotator reviews.

Natural Language Interfaces

• sql_generator.py & pipeline_generator.py (astroml/llm/query): Translates natural language queries to SQL and experiment configurations to runnable YAML.
• validator.py & executor.py (astroml/llm/query): Enforces read-only commands (blocking DROP/DELETE/UPDATE), join limits, and query timeouts with audit logs.

Web Routes & Registrations

• cost.py & query.py (api/routers): REST routers for budget management, usage forecasts, and query translations.
• streaming.py & llm.py (api/routers / api/websocket): Budget-aware SSE and bidirectional WebSockets endpoints.
• app.py & init.py (api / api/routers): Registered new endpoints in the FastAPI app instance.

[drips-wave]

@dorismaduegbunam Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits