Add the "auto-detect the current allocation" feature - #234
Draft
DilumAluthge wants to merge 2 commits into
Draft
Conversation
DilumAluthge
force-pushed
the
dpa/auto_detect
branch
from
February 3, 2025 01:08
076b6b8 to
c012c1f
Compare
Collaborator
|
if we have auto-detect in this package, but certain cluster backends live in separate packages (e.g. LSF), what should user do? I imagine we might want to further split backends into their own packages but then the auto-detect would be even less useful. Is the long-term goal to make this package an umbrella package? If so, maybe this pkg should (optionally) depend on LSF? |
Member
Author
|
All good questions. At this point this PR is definitely very speculative, so everything is still up in the air. At this point I'm not even sure if auto-detect is a good idea. But if we do pursue auto-detect, then I think that yes, ClusterManagers.jl would need to take direct dependencies on LSFClusterManager.jl, SlurmClusterManager.jl, and any other external packages. |
DilumAluthge
force-pushed
the
dpa/auto_detect
branch
5 times, most recently
from
February 10, 2025 00:23
223da63 to
080e0be
Compare
DilumAluthge
force-pushed
the
dpa/auto_detect
branch
from
February 15, 2025 17:00
080e0be to
2777104
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Most of the time, I think that users will only be working with a single cluster scheduler, and thus they can just use the relevant
ClusterManagerdirectly.However, in some situations, I think that it might be useful to be able to write a single script that is agnostic to the specific cluster scheduler. In those cases, it would be nice to auto-detect which cluster scheduler is active, and then automatically use the correct
ClusterManager.This PR adds an experimental non-public
addprocs_autodetect_current_scheduler()function that implements this.Note: The
addprocs_autodetect_current_scheduler()function should be run from inside an active allocation. So e.g. for Slurm, you would first get an allocation (e.g.sbatchorsalloc), and then run this function inside the allocation.