Skip to content

Add the "auto-detect the current allocation" feature - #234

Draft
DilumAluthge wants to merge 2 commits into
masterfrom
dpa/auto_detect
Draft

Add the "auto-detect the current allocation" feature#234
DilumAluthge wants to merge 2 commits into
masterfrom
dpa/auto_detect

Conversation

@DilumAluthge

Copy link
Copy Markdown
Member

Most of the time, I think that users will only be working with a single cluster scheduler, and thus they can just use the relevant ClusterManager directly.

However, in some situations, I think that it might be useful to be able to write a single script that is agnostic to the specific cluster scheduler. In those cases, it would be nice to auto-detect which cluster scheduler is active, and then automatically use the correct ClusterManager.

This PR adds an experimental non-public addprocs_autodetect_current_scheduler() function that implements this.

Note: The addprocs_autodetect_current_scheduler() function should be run from inside an active allocation. So e.g. for Slurm, you would first get an allocation (e.g. sbatch or salloc), and then run this function inside the allocation.

@Moelf

Moelf commented Feb 3, 2025

Copy link
Copy Markdown
Collaborator

if we have auto-detect in this package, but certain cluster backends live in separate packages (e.g. LSF), what should user do?

I imagine we might want to further split backends into their own packages but then the auto-detect would be even less useful.

Is the long-term goal to make this package an umbrella package? If so, maybe this pkg should (optionally) depend on LSF?

@DilumAluthge

Copy link
Copy Markdown
Member Author

All good questions. At this point this PR is definitely very speculative, so everything is still up in the air.

At this point I'm not even sure if auto-detect is a good idea.

But if we do pursue auto-detect, then I think that yes, ClusterManagers.jl would need to take direct dependencies on LSFClusterManager.jl, SlurmClusterManager.jl, and any other external packages.

@DilumAluthge
DilumAluthge force-pushed the dpa/auto_detect branch 5 times, most recently from 223da63 to 080e0be Compare February 10, 2025 00:23
@JuliaParallel JuliaParallel deleted a comment from codecov Bot Sep 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants