Driftmark — End-to-End Churn Prediction + Campaign Scoring (Tabular + NLP + Sequence)

Driftmark is an end-to-end churn modeling project that reads source data from data/raw, builds weekly customer snapshots and churn labels, trains multiple models (tabular, NLP, sequence), ensembles them into a calibrated churn probability, generates explanation-driven “reason codes”, produces a campaign file for activation, serves scores via a FastAPI inference API, and monitors feature drift.

---

Architecture

Three model modalities — tabular, NLP (support tickets), and sequence (usage events) — feed a single calibrated stacker. Each captures a different signal: who they are, what they told you, and what they did.

                    ┌─────────────────────────────────────────────────────┐
 data/raw ─────────►│  [1] snapshots + labels (churn_30d)                │
                    └───────────────┬─────────────────────────────────────┘
                                    ▼
        ┌───────────────────────────────────────────────────────────────┐
        │                    stacker input (per snapshot)               │
        │                                                               │
        │  tabular features ──► LightGBM ─────────────►  p_churn_tab    │
        │  ticket text ──────► MiniLM emb ─► SGD ─────►  p_churn_nlp    │
        │  usage events ─────► 16-week seq ─► LSTM ───►  p_churn_seq    │
        │                                                               │
        └───────────────────────────────┬───────────────────────────────┘
                                        ▼
        ┌───────────────────────────────────────────────────────────────┐
        │  LogisticRegression stacker on [p_tab, p_nlp, p_seq,          │
        │  has_nlp, has_seq, n_tickets_30d]                             │
        │  → Isotonic calibration fit on the validation window only     │
        └───────────────────────────────┬───────────────────────────────┘
                                        ▼
                 p_churn_final → reason codes → campaign.csv → drift report
                 (data/models, data/campaign.csv, data/reports)

Missing modalities are treated as missing, not as zeros: a snapshot with no tickets gets p_churn_nlp filled with the train base rate and a has_nlp flag so the stacker learns to discount the fill. Splits and calibration are time-aware (no temporal leakage). Full write-up: docs/architecture.md.

---

Quickstart (Windows / PowerShell)

1) Install dependencies

poetry install

2) Run the full pipeline (one command)

poetry run python -m retainflow.pipeline.run_all

This will attempt to run, in order:

• raw-data validation (data/raw/*.parquet)
• weekly snapshot generation (data/splits/snapshots.parquet)
• churn label generation from data/raw/churn.parquet
• tabular feature building (data/features/features_tabular.parquet)
• tabular training
• NLP training (ticket text → snapshot-level risk)
• sequence training
• ensemble training + calibration (time-aware; calibrates on validation window)
• tabular explanation generation (LightGBM SHAP contributions via pred_contrib=True)
• batch campaign scoring (data/campaign.csv)
• drift report (data/reports/drift_report.json)

You can also resume from a step:

poetry run python -m retainflow.pipeline.run_all --from-step ensemble

---

Source data

The pipeline treats data/raw as the authoritative source layer. Required files:

• data/raw/customers.parquet
• data/raw/subscriptions.parquet
• data/raw/usage_events.parquet
• data/raw/support_tickets.parquet
• data/raw/churn.parquet

churn_30d labels are built directly from churn.parquet by checking whether a customer churns within 30 days after each snapshot date.

---

Run the API

Start server

poetry run uvicorn retainflow.inference.api:app --reload

Swagger UI

Open:

• http://127.0.0.1:8000/docs

Endpoints

• GET /health
  • Returns service status and whether prediction/explanation artifacts are available.
• GET /overview
  • Returns the latest scoring snapshot summary, risk distribution, top reason codes, and recent backend activity.
• GET /score?customer_id=123&as_of_date=YYYY-MM-DD
  • Returns churn probability + segment + reason codes + recommended action + snapshot metadata.
  • If as_of_date is omitted, API uses the latest available date in predictions.
• POST /score_batch
  • Body:

    {
      "customer_ids": [1, 2, 3],
      "as_of_date": null
    }

  • Returns an array of scores for the requested customers.
• GET /campaigns
  • Returns the current campaign export, segment breakdown, and top reasons.
• POST /campaigns
  • Regenerates the campaign export from backend predictions.
• GET /monitoring/drift
  • Returns the latest drift report plus table/chart-ready metrics.
• GET /artifacts
  • Returns current model/artifact metadata for the frontend settings page.

---

Generate campaign file (batch scoring)

Produces a CSV for downstream activation/CRM.

poetry run python -m retainflow.inference.batch_score --top-k 50000 --out data/campaign.csv

Outputs:

• data/campaign.csv
• data/campaign_meta.json

Note: CSV can’t store nested data. List-like explanation columns are flattened to pipe-delimited strings.

---

Explanations (“Reason codes”)

Tabular explanations are produced from LightGBM feature contributions.

poetry run python -m retainflow.models.explain_tabular

Outputs:

• data/models/explanations_tabular.parquet
  • customer_id, as_of_date
  • reason_codes (List[str])
  • reason_features (List[str])
  • reason_contribs (List[float])
• data/models/explanations_tabular_meta.json

These are joined into batch_score and optionally returned by the API.

---

Drift monitoring

Generates a drift report comparing baseline vs latest distributions:

poetry run python -m retainflow.monitoring.drift_report

Outputs:

• data/reports/drift_report.json

Includes:

• PSI per numeric feature
• missing-rate changes
• ok / warn / alert flags

---

Model artifacts & outputs

Common output files:

Predictions

• data/models/pred_ensemble.parquet
  • p_churn_final (calibrated)
  • p_churn_raw (pre-calibration, if enabled)
• data/models/pred_nlp.parquet
  • p_churn_nlp
  • n_tickets_30d (evidence strength signal)
• data/models/pred_seq.parquet (if sequence model is enabled)

Ensemble

• data/models/ensemble_metrics.json
• data/models/ensemble_stacker_artifacts.npz
  • logistic stacker weights
  • isotonic calibration thresholds
  • base churn rate used for missing-modality fills

Campaign output

• data/campaign.csv
• data/campaign_meta.json

Monitoring

• data/reports/drift_report.json

---

Testing

Install dev deps (if needed):

poetry add --group dev pytest httpx

Run tests:

poetry run pytest -q

Tests include:

• artifact existence + probability sanity checks
• FastAPI smoke tests (/health, /score)

---

Evaluation

Reproduce the reported numbers on demand:

poetry run python scripts/evaluate.py

scripts/evaluate.py recomputes the ensemble's held-out test window (the final ~15% of as_of_date values, the same time split used by retainflow.models.ensemble) and prints:

• test ROC AUC / PR AUC, F1 at the 0.5 threshold, and the best-F1 threshold
• which modalities were present (tabular / NLP / sequence)
• the drift summary and campaign metadata

Current recorded test-window metrics (data/models/ensemble_metrics.json):

Model	Test ROC AUC	Test PR AUC
Tabular LightGBM (lgbm_tabular_metrics.json)	~0.886	~0.610
Ensemble (calibrated stacker)	~0.888	~0.576

Live checkpoints: GET /overview (risk distribution + top drivers), GET /artifacts (metric + artifact status), and the campaign export at GET /campaigns/download.

Campaign output sample (data/campaign.csv):

customer_id,as_of_date,p_churn_final,segment,reason_codes,recommended_action
141679,2025-12-29,1.0,high_risk,usage_decline|inactive_recency|billing_risk|support_friction|usage_decline,priority_support_outreach
142107,2025-12-29,1.0,high_risk,inactive_recency|billing_risk|usage_decline|usage_decline|usage_decline,autopay_nudge
16033,2025-12-29,0.9787234,high_risk,usage_decline|usage_decline|billing_risk|usage_decline|usage_decline,autopay_nudge
33044,2025-12-29,0.9787234,high_risk,usage_decline|usage_decline|billing_risk|usage_decline|usage_decline,autopay_nudge
72917,2025-12-29,0.9787234,high_risk,inactive_recency|usage_decline|usage_decline|billing_risk|usage_decline,autopay_nudge

---

Deployment

The included Dockerfile builds a self-contained API image: it generates a small demo dataset, trains the tabular + ensemble models, produces explanations, the campaign export and a drift report, then serves the FastAPI app — so /score, /overview, /campaigns, and /monitoring/drift all return real data with zero local setup.

docker build -t driftmark-api .
docker run -p 8000:8000 driftmark-api

Hosting:

• Railway (recommended, ~5 min): railway.json is already in the repo and auto-detects the Dockerfile builder.
  1. git push this branch to GitHub.
  2. On railway.app → New Project → Deploy from GitHub repo → pick Driftmark.
  3. Railway builds the image (bakes the demo artifacts) and starts the API. Done.
• Render: new Web Service → connect the GitHub repo → build method Dockerfile. No extra config needed.
• Fly.io: flyctl launch from the repo root and accept the defaults.

Live deployment: TBD — set after your first deploy. Once Railway/Render/Fly gives you a URL, replace https://driftmark-api.up.railway.app in this README with it.

If you deploy the Vite frontend (frontend/), set the API's CORS allow-list via the CORS_ALLOW_ORIGINS environment variable (comma-separated), e.g. CORS_ALLOW_ORIGINS="https://app.yoursite.com". Defaults to localhost:5173.

---

Notes / Implementation choices

• Authoritative input layer: The project runs from data/raw forward. Derived outputs in data/splits, data/features, data/models, and data/reports can be regenerated.
• Labeling: churn_30d is generated directly from data/raw/churn.parquet, not from synthetic rules.
• Time-aware calibration: The ensemble stacker is trained on a training window; calibration is fit on a validation window only (no temporal leakage).
• Missing modalities: If NLP/sequence predictions are missing for a snapshot, probabilities are filled with the train base rate, and binary availability flags are provided so the stacker learns reliability.
• SHAP explanations: Uses LightGBM pred_contrib=True for fast per-row feature contributions (no additional SHAP dependency).
• Windows stability: Parquet is used for large artifacts; CSV is reserved for the final campaign file with flattened fields.

---

Troubleshooting

“Missing ensemble predictions”

Run:

poetry run python -m retainflow.models.ensemble

“No score found for that customer_id/as_of_date”

Use the latest date or omit as_of_date:

• /score?customer_id=9

Polars CSV nested errors

Nested columns must be flattened or dropped before CSV output. The batch scoring script already handles this automatically.