Distributed LLM Request Processing System

A high-performance, asynchronous backend designed to handle large-scale prompt requests with durable execution, semantic caching, and strict rate-limit compliance.

System Architecture

The system utilizes a distributed task queue architecture to decouple request ingestion from heavy processing.

Tech Stack

• FastAPI: Asynchronous API framework for high-concurrency request ingestion.

• Celery: Distributed task queue for parallel prompt processing and retries.

• Redis: Message broker for task distribution and inter-process communication.

• PostgreSQL + pgvector: Unified storage for relational metadata and high-dimensional vector embeddings.

• Sentence-Transformers: Local all-MiniLM-L6-v2 model for deterministic embedding generation.

---

Getting Started

The entire stack is fully containerized. No manual database setup or migrations are required.

One-Command Setup

Bash

docker-compose up --build

The system will automatically:

1. Initialize the PostgreSQL schema and enable the vector extension.

2. Pre-download the Sentence-Transformer model into the worker image.

3. Expose the API on http://localhost:8000.

---

Features & Endpoints

1. Health Check

Verify the connection status of the Database, Redis broker, and Celery worker.

Bash

curl http://localhost:8000/health

2. Process Prompt

Submit a prompt for asynchronous processing.

Bash

curl -X POST http://localhost:8000/process\
-H "Content-Type: application/json"\
-d '{
  "user_id": "intern_user_01",
  "prompt_id": "p_1001",
  "text": "What are the core principles of distributed systems?",
  "priority": "high"
}'

3. Semantic Caching

The system uses pgvector to identify semantically similar prompts (Similarity > 0.9). If a match is found, the cached response is returned, bypassing the LLM call.

Bash

curl -X POST http://localhost:8000/process\
-H "Content-Type: application/json"\
-d '{
  "user_id": "intern_user_01",
  "prompt_id": "p_1002",
  "text": "Tell me the main principles of distributed systems",
  "priority": "normal"
}'

Note: The second request will return "cached": true.

---

Design & Resilience

Durable Execution

If a worker crashes mid-task, the task remains in the Redis queue. Upon restart, the worker will automatically pick up the pending task and resume processing.

Rate Limiting

The MockLLM service implements a sliding window rate limiter to strictly adhere to the 300 requests/minute provider constraint across all parallel workers.

Database Strategy

Chosen for a 3:1 write-to-read ratio. Every prompt is persisted immediately to ensure durability. We use pgvector's <=> (Cosine Distance) operator for high-speed similarity lookups.

---

Testing

Resilience Test

To verify the system's ability to recover from worker failure:

Bash

chmod +x scripts/test_resilience.sh
./scripts/test_resilience.sh

Manual DB Inspection

Inspect the state of your prompts, embeddings, and hit counts:

Bash

docker exec -it request-processing-backend-db-1 psql -U user -d llm_db -c "SELECT prompt_id, status, hit_count, processing_time_ms FROM prompt_requests;"

---

Repository Structure

Plaintext

.
├── main.py              # API Entry point
├── celery_app.py        # Celery configuration
├── docker-compose.yml   # Infrastructure orchestration
├── Dockerfile           # Multi-stage build for API/Worker
├── api/                 # API Routes & logic
├── core/                # Database config & global settings
├── models/              # SQLAlchemy & Pydantic schemas
├── services/            # MockLLM & Embedding logic
├── worker/              # Celery task definitions
└── scripts/             # Resilience & Load testing scripts