Skip to content

feat: add NCCL timeout config, stale ZMQ socket cleanup, OmegaConfig resolvers#2862

Open
puneeshkhanna wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
puneeshkhanna:feat/utils
Open

feat: add NCCL timeout config, stale ZMQ socket cleanup, OmegaConfig resolvers#2862
puneeshkhanna wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
puneeshkhanna:feat/utils

Conversation

@puneeshkhanna

Copy link
Copy Markdown

What does this PR do ?

  • Add configurable NCCL collective timeout via NRL_DIST_TIMEOUT_MINUTES env var (default 10 mins) to handle slow cross-node collectives during HF->Megatron reshard and policy->vLLM refit.
  • Self-heal stale IPC socket files in maybe_init_zmq() to prevent EADDRINUSE errors after unclean exits (crash, kill -9, OOM).
  • Register "add" and "sub" OmegaConf resolvers for use in config files. Helps to have below auto calculations in config files for agent harnesses.
    max_input_tokens: ${sub:${policy.generation.vllm_cfg.max_model_len}, ${policy.generation.max_new_tokens}}
    max_output_tokens: ${policy.generation.max_new_tokens}

Issues

List issues that this PR closes (syntax):

Usage

  • You can potentially add a usage example below
# Add a code snippet demonstrating how to use this

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
  • Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Additional Information

New tests not required.

/review-pr comments below:
Files changed: 3

Findings (scored ≥80)

None. LGTM.

Filtered (scored <80)

All candidates were either consistent with codebase conventions (env var ValueError — 92% confidence it's a non-issue) or pre-existing gaps (missing resolver tests). No
new bugs or guideline violations introduced by this diff.

The changes are clean and ready.

…f resolvers

- Add configurable NCCL collective timeout via NRL_DIST_TIMEOUT_MINUTES env var
  (default 10 mins) to handle slow cross-node collectives during HF->Megatron
  reshard and policy->vLLM refit.
- Self-heal stale IPC socket files in maybe_init_zmq() to prevent EADDRINUSE
  errors after unclean exits (crash, kill -9, OOM).
- Register "add" and "sub" OmegaConf resolvers for use in config files.

Signed-off-by: Puneesh Khanna <puneesh.khanna@tii.ae>
@puneeshkhanna puneeshkhanna requested review from a team as code owners June 17, 2026 07:49
@copy-pr-bot

copy-pr-bot Bot commented Jun 17, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@svcnvidia-nemo-ci svcnvidia-nemo-ci added the waiting-on-maintainers Waiting on maintainers to respond label Jun 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-request waiting-on-maintainers Waiting on maintainers to respond

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants