Feature/llm observability search stream cache

[devfoma]

LLM Platform Observability, Search, Streaming, and Caching

This PR implements four major features supporting LLM operations on the AstroML platform: a real-time monitoring dashboard, a unified hybrid semantic search engine, high-performance token streaming endpoints, and intelligent exact/semantic caching layers.

Core Features Implemented

1. LLM Monitoring Dashboard ([FEATURE] LLM Monitoring Dashboard — Real-time observability for LLM operations #463)

   • In-memory metrics collection and statistical analysis for requests/min, tokens/sec, success/error rates, latency percentiles, and cost tracking.
   • Real-time alert definitions and rule validation (e.g. latency, error rate, spend threshold alerts).
   • Prometheus exporter integration for scraping performance metrics.
   • Comprehensive frontend dashboard visual component built with Recharts.

2. Semantic & Hybrid Search ([FEATURE] Semantic Search — Natural language search across all platform data #462)

   • Core search coordinator combining BM25 keyword frequencies with vector similarity matching.
   • Automated indexing pipeline for platform data sources (Docs, Transactions, Alerts, Models, and Code files).
   • Cross-encoder reranking strategy for precise query results.
   • Unified search and autocomplete suggestion endpoints.

3. LLM Streaming Service ([FEATURE] LLM Streaming Service — Real-time token streaming API #461)

   • Async generator-based streaming architecture yielding tokens in real-time.
   • Adaptive buffering with backpressure compensation to scale token delivery efficiency dynamically.
   • Reconnection manager for session persistence, heartbeat monitoring, and recovery of interrupted streams.
   • Full support for Server-Sent Events (SSE) and WebSocket full-duplex channels.

4. LLM Caching Service ([FEATURE] LLM Caching Service — Semantic and exact-match caching for LLM responses #460)

   • Redis and PostgreSQL caching backends with LRU/LFU eviction policies.
   • Semantic similarity cache-key matching utilizing cosine similarity thresholds.
   • Pre-warming strategy configured for common query templates.
   • Redis monitoring dashboard layout.

closes #463
closes #462
closes #461
closes #460

[drips-wave]

@devfoma Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits