Atlas Data Platform

Atlas is an end-to-end scalable data ingestion pipeline and API service. It is designed to handle high-throughput financial transactions, customer data, and streaming events, ensuring that all data is validated, processed, and reliably stored in a PostgreSQL data warehouse. This project is built following the Medallion Architecture (Bronze -> Silver -> Gold).

System Architecture

The platform architecture handles both real-time streaming and robust batch workloads, decoupling ingestion from processing and orchestrating Medallion architecture transformations (Bronze -> Silver -> Gold).

graph TD
    subgraph "Ingestion Layer"
        Kafka[Apache Kafka<br>Real-time Events]
        Batch[Batch Files<br>CSV / ERP Data]
        API[FastAPI Endpoints<br>REST Requests]
    end

    subgraph "Orchestration & Processing (Python 3 / Airflow)"
        Airflow[Apache Airflow DAG<br>Pipeline Orchestrator]
        Consumer[Kafka Consumer<br>Streaming Pipeline]
        Reader[Batch Reader &<br>PostgresLoader]
        Router[API Routers<br>Sales & DQ]
        
        Validator[Data Quality &<br>Validation Rules]
        
        Airflow --> Reader
        Airflow --> Validator
        Consumer --> Validator
        Reader --> Validator
        Router --> Validator
    end

    subgraph "Storage & Transformation (Medallion)"
        PG[(PostgreSQL Warehouse<br>Bronze / Silver / Gold)]
    end

    subgraph "Observability & Infrastructure"
        Prom[Prometheus & Grafana<br>Telemetry Dashboards]
        K8s[Kubernetes<br>Manifests & HPA]
        TF[Terraform<br>Infrastructure as Code]
    end

    Kafka --> Consumer
    Batch --> Reader
    API --> Router
    
    Validator --> PG
    PG --> Prom

Loading

Core Components

• Streaming Pipeline: Consumes real-time financial and operational events from Apache Kafka. Built with resilience in mind, featuring graceful degradation, structured logging, and idempotent processing to prevent data duplication.
• Batch Ingestion & Orchestration: Implements Apache Airflow DAGs (orchestration/dags/daily_ingestion_dag.py) to schedule and orchestrate end-to-end batch ingestion, triggering dbt transformations across Bronze, Silver, and Gold layers.
• High-Performance APIs: Built with FastAPI. Features asynchronous database sessions, secure dependency injection, and scalable routing (handling both Data Quality metrics and Sales data).
• Data Quality Engine: A suite of validation rules and reporting mechanisms. Ensures data hygiene before persistence, applying business rules (e.g., verifying transaction dates and numeric thresholds) dynamically.
• Infrastructure & Container Orchestration: Fully configured with Kubernetes manifests (infra/k8s/) including Deployments, Services, Horizontal Pod Autoscalers (HPA), and Alembic DB Migration Jobs, alongside Terraform (infra/terraform/) resources.
• Observability: Prometheus metrics and Grafana dashboards (infra/grafana/dashboards/atlas_overview.json) tracking ingestion throughput, request latency, and DQ failure rates.

Tech Stack

This project was built with a strong emphasis on reliability, scalability, and code quality:

• Language & Frameworks: Python 3.11+, FastAPI, SQLAlchemy, dbt
• Data & Streaming: PostgreSQL, Apache Kafka, Apache Airflow
• Infrastructure & Deployment: Kubernetes (K8s), Docker, Terraform, Helm
• Observability: Prometheus, Grafana
• Test-Driven Development (TDD): Comprehensive test suite utilizing pytest and Docker-based testcontainers. The project maintains >97% code coverage across unit and integration tests.
• Tooling: uv for dependency management, ruff and black for strict static analysis and formatting.

Getting Started

Prerequisites

• Python 3.11+
• Docker (for integration testing with Testcontainers)
• uv (Fast Python package and project manager)

Installation & Setup

1. Install dependencies:

   uv sync

2. Run formatting and linting checks:

   uv run black src/ tests/
   uv run ruff check src/ tests/

3. Run the test suite: Execute all unit and integration tests with coverage reporting:

   uv run pytest tests/ --cov=src/atlas --cov-report=term-missing

4. Start the API server:

   uv run uvicorn atlas.api.main:app --reload

5. Apply Kubernetes Manifests:

   kubectl apply -f infra/k8s/

Future Improvements

While Atlas is built with robust production-ready patterns, future iterations could expand the pipeline topology:

• Dead Letter Queues (DLQ): Implementing a DLQ topology in Kafka to allow data engineers to inspect, fix, and safely replay rejected payloads without data loss.
• Multi-Region Disaster Recovery: Expanding Terraform definitions for multi-region cloud PostgreSQL replication and global Kafka cluster mirroring.