Repository navigation

[Python][RunInference] Add a capacity-based autoscaling signal for model-server handlers - #40907

Open

AMOOOMA wants to merge 14 commits into

apache:masterfrom

AMOOOMA:pr-model-server-autoscaling

Open

[Python][RunInference] Add a capacity-based autoscaling signal for model-server handlers#40907
AMOOOMA wants to merge 14 commits into
apache:masterfrom
AMOOOMA:pr-model-server-autoscaling

Commits on Oct 30, 2026
- [RunInference] Add a capacity-based autoscaling signal for model-server handlers
  AMOOOMA
  committed
  6fd3db5
Commits on Nov 2, 2026
- Clean up docstrings and readability in autoscaling_metrics
  AMOOOMA
  committed
  3610dd1
Commits on Nov 3, 2026
- Make NumLoadedModelsPublisher a plain sink
  AMOOOMA
  committed
  f84e5f4
- Package the model-server capacity signal behind one call
  AMOOOMA
  committed
  4345b29
- Fold the sampler and signal glue into make_model_server_publisher
  AMOOOMA
  committed
  833afa7
- Make AutoscalingMetricConfig plain data
  AMOOOMA
  committed
  5085d0d
- Let NumLoadedModelsPublisher read the server itself
  AMOOOMA
  committed
  746051e
- Describe the capacity signal in requests and state its units
  AMOOOMA
  committed
  098fd25
- Add completion-rate aggregation modes to the capacity estimator
  AMOOOMA
  committed
  3a2d345
- Tidy the estimator docstring and fill test gaps
  AMOOOMA
  committed
  a84f728
- Publish the autoscaling metric atomically and time samples monotonically
  AMOOOMA
  committed
  2cd0ec0
Commits on Nov 5, 2026