Skip to content

Parallel auto and timing order - #18

Closed
danielmarbach wants to merge 6 commits into
mainfrom
parallel-auto-and-timing-order
Closed

Parallel auto and timing order#18
danielmarbach wants to merge 6 commits into
mainfrom
parallel-auto-and-timing-order

Conversation

@danielmarbach

@danielmarbach danielmarbach commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

No description provided.

@danielmarbach
Add max-parallel: auto to match the runner's CPU count
ad36c7a
@danielmarbach
Start the slowest parallel runs first when durations are available
5868ddc 
Durations of successful runs are read from and written to TIMINGS_FILE.
Runs without a recorded duration start first; without data the order
stays alphabetical.
@danielmarbach
Cache test run durations between invocations
a393382 
Restores and saves the timings file with actions/cache so the next run
can start the slowest assemblies first. Adds cache-timings and
timings-cache-key inputs.
@danielmarbach
Cap max-parallel at the number of runs
cbc1607
@danielmarbach
Average recorded durations with the previous value
7b44ea4 
One noisy run no longer reshuffles the start order of the next.
@danielmarbach
Keep the last 10 durations per run and order by their median
1c9082d 
Replaces the pairwise average so the order follows recent history rather
than the last two runs. Entries holding a single number from earlier
versions are read as one sample.
@danielmarbach

danielmarbach commented Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

Tested in Particular/NServiceBus.Transport.AzureServiceBus#1524 and Particular/ServiceControl#5966

@danielmarbach danielmarbach mentioned this pull request Oct 8, 2026
Merged
@danielmarbach

danielmarbach commented Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

Closing, the experiment isn't worth pursuing.

I tried max-parallel: auto and starting the slowest assemblies first from durations cached between runs, and compared them against ServiceControl's master over several CI runs.

  • No reproducible gain. On ServiceControl, no leg improved beyond what normal run-to-run variation explains. The best case, Windows-Raven at 305 s against a 396 s median, did not repeat (469 s and 422 s in later runs with the same ordering).
  • Often already optimal. Some legs already sit at their longest assembly's duration (Linux-Raven), and RabbitMQ has as many slots as assemblies, so ordering can't help there.
  • Can backfire. Starting the heaviest assemblies together on a shared SQL Server container made Windows-SqlServer slower, at 390–414 s timed against 314 s alphabetical.
  • The signal is too noisy to schedule on. Identical GitHub-hosted runners vary a lot between builds (Akinshin), and master itself varies by up to ±20% on some legs.
  • auto has no use here. In Azure Service Bus tests the assemblies mostly wait on I/O, so CPU count is the wrong limit. max-parallel: 6 instead of 4 cut that job from about 365 s to about 235 s.

The max-parallel validation part is in #19.

@danielmarbach
danielmarbach deleted the parallel-auto-and-timing-order branch October 8, 2026 10:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@danielmarbach danielmarbach

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@danielmarbach