Is Highs::resetGlobalScheduler() really not safe to call while other solves run in any thread.

[soppera]

Highs::resetGlobalScheduler() documentation clearly states:

It is not thread-safe to call this function while calling run() or presolve() on any other Highs instance in any thread.

But this comment has been written on change 8b7f996, before change 3728cbb landed and added a thread_local executor.

It seems the C API (highs/interfaces/highs_c_api.cpp) also calls highs.resetGlobalScheduler(true) at the end of each solve.

So my question is: is the documentation of resetGlobalScheduler() still correct, or it is now safe to call this function on the thread that instantiated a Highs and used it to solve a model even if other threads may have instantiated other Highs?

[jajhall]

Please would you comment @filikat or @galabovaa

[filikat]

You're right!
I had to look again at the parallel scheduler code because my understanding was slightly wrong (I thought the scheduler was static).
Everything related to the scheduler is thread_local, so if a thread creates a Highs instance and initialises the scheduler, the scheduler can be accessed only by that thread and any worker thread created by it during initialisation of HighsTaskExecutor.
If a separate thread (not created by Highs, but created externally by the user) creates a Highs instance, that thread gets its own parallel scheduler, independent of the first one.
Calling Highs::resetGlobalScheduler then kills only the scheduler associated with the thread that calls it. The scheduler in the other thread continues existing.

Calling Highs::resetGlobalScheduler inside a worker thread created by Highs, while another worker thread is still running tasks, is not safe. However, it is safe to call it from the main thread that created the scheduler, when no other tasks are running in that scheduler, as this call does not affect the scheduler owned by other threads (created independently of the Highs scheduler).

We removed the call to Highs::resetGlobalScheduler from the C api solve functions and from Highs_destroy in the latest branch, because it was creating issues when running many tests in parallel in Julia. I'm now not sure any more what the issue was there (cc @odow).

This also means that within the same process it is possible to have multiple Highs instances running in parallel (again, parallelism obtained independently of the Highs scheduler), each using a different number of threads (I left wrong comments that now need updating in #3141). If these parallel Highs instances rely on the default number of threads, it is quite likely that there will be performance degradation due to too many threads being active at the same time.

[filikat]

We definitely need to document the behaviour of the scheduler better.
Here's a possibility (assisted by Claude):

Parallel scheduler: initialization, shutdown and thread safety

Terminology

• Application thread: a thread you create yourself (with std::thread, OpenMP, pthreads, Julia, Python, and so on), or the main thread of your program.
• Scheduler: the pool of threads HiGHS uses to run parallel work.
• Worker thread: a thread created by HiGHS when it creates a scheduler. Worker threads only run HiGHS tasks.
• Owning thread: the application thread that caused a scheduler to be created. It takes part in the parallel work together with the worker threads.

Each application thread has its own scheduler

A scheduler is tied to the application thread that created it, not to a Highs instance.

• The first time an application thread calls run() or presolve(), HiGHS creates a scheduler for that thread. The thread becomes its owning thread.
• All Highs instances used on the same application thread share that thread's scheduler.
• Application threads never share a scheduler. If several application threads each use HiGHS, each gets its own scheduler and its own worker threads. They do not interfere with each other.
• The number of threads is fixed when the scheduler is created. Changing the threads option later does not resize an existing scheduler.

Because every scheduler adds worker threads, running HiGHS from many application threads with the default thread count can create far more threads than you have cores, which slows everything down. In that case set the threads option explicitly.

Shutdown

A scheduler is shut down when its owning thread exits, or when you call Highs::resetGlobalScheduler(). Without an explicit call, a scheduler stays alive between solves, so that it can be reused. Call resetGlobalScheduler(true) if you want to free its threads and memory sooner, or to wait for the worker threads to finish.

Resetting only affects the scheduler of the calling thread. If the calling thread has no scheduler, the call does nothing. Other threads' schedulers are not affected.

Thread safety

Call resetGlobalScheduler() only:

• from the thread that owns the scheduler (the one that ran the solve), and
• when no run() or presolve() is in progress on that thread, including from inside a callback.

Do not call it from inside a HiGHS parallel task or from a worker thread. That would stop the whole scheduler while other workers may still be running tasks.

After a reset, the next run() or presolve() on that thread creates a new scheduler.

Using the C API

The C API follows the same rules. Each application thread that calls Highs_run (or any other function that solves a model) gets its own scheduler, and Highs_destroy does not shut it down. Instances created and used on the same thread share that thread's scheduler. To control the number of threads, set the threads option with Highs_setIntOptionValue(highs, "threads", n) before the first solve on a thread. To free the scheduler explicitly, call Highs_resetGlobalScheduler(1) to wait for the worker threads to finish, or Highs_resetGlobalScheduler(0) to return immediately. The same restrictions as in C++ apply: call it from the thread that ran the solve, when no solve is in progress on that thread (including from inside a callback), and never from a worker thread or a HiGHS task.

[soppera]

when no run() or presolve() is in progress on that thread, including from inside a callback.

The "including from inside a callback" is a bit weird as this is the only way to call resetGlobalScheduler() on the same thread as the one having a pending call to run()/presolve(). The thread being blocked on run()/presolve(), it is not possible for the user to call any other function on this thread except if Highs calls them back.

Note that there would be the potential case of a signal handler, but I don't think any user would assume this API to be signal-safe or call it from a signal handler.

[filikat]

Fair point. I will change wording.

[jajhall]

Good catch everyone.

Although we must keep the name 'resetGlobalScheduler' for back-compatibility, to me it sounds wrong now, since it's associated with a particular application thread. Perhaps 'resetGlobalScheduler' could be renamed something like 'resetApplicationScheduler', with a method 'resetGlobalScheduler' that just calls 'resetApplicationScheduler'.

Otherwise this statement reads well - and is comprehensible! - to me

[soppera]

Thanks for the confirmation.

I was asking about this as in https://github.com/google/or-tools/blob/7e7ecbff53afee3ff556f928998dcae99f62abb0/ortools/math_opt/solvers/highs_solver.cc#L226-L233 we explicitly disallow changing the number of threads because we assumed there was no thread-safe way of doing it.

I'll change our code.