ci: Add estimator regression CI

[AdilZouitine]

What & Why

This PR adds an estimator regression CI suite for public ML estimators. The goal is to catch behavior changes that regular unit tests can miss: small metric regressions, scenario errors, removed coverage, and accidental changes to a scenario's workload.

The suite intentionally recomputes results for both main and the PR branch inside the CI run instead of comparing the PR against a stored main artifact. A stored artifact can become stale when the CI image, Python patch version, dependencies, or uv.lock change. Running both sides in the same workflow keeps the comparison tied to the current CI environment and makes the report easier to trust.

The suite runs deterministic workloads on the base branch and the PR branch, writes YAML metrics artifacts for that run, compares the two, uploads the artifacts, and posts a Markdown report on the PR. No linked issue.

How

The implementation keeps the system intentionally boring: scenarios are plain Python entries with River constructors, workloads are deterministic Python functions, and YAML is only used for run artifacts.

The first commit adds the estimator regression runner, comparator, workload definitions, scenario inventory, Make targets, dependency updates, and the GitHub Actions workflow. The second commit adds the README that documents local usage, the coverage model, the determinism contract, and the comparison policy.

The workflow has two jobs:

• inventory audits that every concrete public estimator is either covered by a scenario or explicitly excluded with a reason.
• run checks out the PR branch and the base branch in the same job, installs each checkout from its lockfile, runs the suite on both, compares metrics with per-metric tolerances, uploads the fresh artifacts, and comments with the generated report.

This avoids maintaining a permanent baseline artifact for main. The artifacts are still useful, but only as evidence for the current CI run: metrics.base.yml, metrics.head.yml, comparison.yml, and report.md are uploaded so reviewers can inspect exactly what was compared.

The higher risk areas are the scenario inventory and metric tolerances. The inventory decides what behavior is covered. The tolerances decide what CI treats as a real regression instead of expected numerical noise.

Testing

Validated locally with the estimator regression CLI:

• uv run python -m benchmarks.estimator_regression.cli audit
  Expected output: 145 discovered, 90 covered, 55 excluded, 0 problems.
• PYTHONHASHSEED=0 uv run python -m benchmarks.estimator_regression.cli run --output /tmp/er_full_a.yml
  Expected output: all 90 scenarios pass, around 32 seconds locally.
• Ran the full suite twice with PYTHONHASHSEED=0 and compared the YAML files with diff.
  Expected output: no diff, the artifacts are byte-identical for the same commit.
• uv run python -m benchmarks.estimator_regression.cli compare --base /tmp/er_full_a.yml --head /tmp/er_full_b.yml --output /tmp/er_report.md
  Expected output: clean comparison across 90 scenarios and 251 metrics.
• Manually injected a metric regression and removed one metric from a copied artifact.
  Expected output: compare fails and the report points at the regressed and missing metric.

Notes for reviewers

Please review this as a CI signal quality change more than a benchmark implementation. The important questions are whether the covered scenarios are representative enough, whether the exclusions are honest, and whether the tolerances are strict enough to catch real behavior drift without creating noisy failures.

The suite is intentionally limited to ML estimators. Transformers, feature extractors, preprocessors, composition helpers, and search structures are excluded because they need different invariants than predictive metrics.

A useful review path is to start with benchmarks/estimator_regression/README.md, then inspect scenarios.py for coverage choices, run.py for harness behavior, compare.py for gating logic, and finally .github/workflows/estimator-regression.yml for the CI wiring.

AI Usage

Cursor was used to review and simplify the design, identify unnecessary complexity. The final implementation and validation were reviewed against the actual branch diff and local command output.

[github-actions]

Estimator regression: passed

Compared 90 scenarios and 251 metrics.
0 metric regressions beyond tolerance.
0 scenarios errored.
0 scenarios are new and not yet gated.

Artifacts

• metrics.base.yml
• metrics.head.yml
• Download artifact