Skip to content

[chore] Exclude nixl-cu13 from packages#1756

Merged
SumanthRH merged 1 commit into
mainfrom
pin-nixl
Jun 7, 2026
Merged

[chore] Exclude nixl-cu13 from packages#1756
SumanthRH merged 1 commit into
mainfrom
pin-nixl

Conversation

@SumanthRH

Copy link
Copy Markdown
Member

What does this PR do?

Excludes nixl cu13 from the packages.

After #1601 , vllm model resolution fails with

, ip=10.0.53.33, actor_id=4bb0516283686471f260700b04000000, repr=<skyrl.backends.skyrl_train.inference_servers.vllm_server_actor.VLLMServerActor object at 0x706173dea0f0>)
  File "/home/ray/anaconda3/lib/python3.12/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/home/ray/anaconda3/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/ray/session_2026-06-07_06-36-06_373207_3903/runtime_resources/working_dir_files/s3_anyscale-production-data-cld-hxkifz7xa22mwicp21nzkds1lw_org_xc6lv84h3d7m9dljcc17esfw2i_cld_hxkifz7xa22mwicp21nzkds1lw_runtime_env_packages_pkg_78d2d462c60fbf0f44597ff416ba6c12/skyrl/backends/skyrl_train/inference_servers/vllm_server_actor.py", line 279, in start
    await self._wait_until_healthy()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/ray/session_2026-06-07_06-36-06_373207_3903/runtime_resources/working_dir_files/s3_anyscale-production-data-cld-hxkifz7xa22mwicp21nzkds1lw_org_xc6lv84h3d7m9dljcc17esfw2i_cld_hxkifz7xa22mwicp21nzkds1lw_runtime_env_packages_pkg_78d2d462c60fbf0f44597ff416ba6c12/skyrl/backends/skyrl_train/inference_servers/vllm_server_actor.py", line 294, in _wait_until_healthy
    raise exc
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/ray/session_2026-06-07_06-36-06_373207_3903/runtime_resources/working_dir_files/s3_anyscale-production-data-cld-hxkifz7xa22mwicp21nzkds1lw_org_xc6lv84h3d7m9dljcc17esfw2i_cld_hxkifz7xa22mwicp21nzkds1lw_runtime_env_packages_pkg_78d2d462c60fbf0f44597ff416ba6c12/skyrl/backends/skyrl_train/inference_servers/vllm_server_actor.py", line 335, in _run_server
    self._engine = AsyncLLMEngine.from_engine_args(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ray/.cache/uv/builds-v0/.tmpHNSj9v/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 242, in from_engine_args
    vllm_config = engine_args.create_engine_config(usage_context)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ray/.cache/uv/builds-v0/.tmpHNSj9v/lib/python3.12/site-packages/vllm/engine/arg_utils.py", line 1627, in create_engine_config
    model_config = self.create_model_config()
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ray/.cache/uv/builds-v0/.tmpHNSj9v/lib/python3.12/site-packages/vllm/engine/arg_utils.py", line 1475, in create_model_config
    return ModelConfig(
           ^^^^^^^^^^^^
  File "/home/ray/.cache/uv/builds-v0/.tmpHNSj9v/lib/python3.12/site-packages/pydantic/_internal/_dataclasses.py", line 121, in __init__
    s.__pydantic_validator__.validate_python(ArgsKwargs(args, kwargs), self_instance=s)
pydantic_core._pydantic_core.ValidationError: 1 validation error for ModelConfig
  Value error, Model architectures ['Qwen3MoeForCausalLM'] failed to be inspected. Please check the logs for more details. [type=value_error, input_value=ArgsKwargs((), {'model': ...nderer_num_workers': 1}), input_type=ArgsKwargs]
    For further information visit https://errors.pydantic.dev/2.13/v/value_error

The root cause is the nixl package. The nixl package on PyPI is a meta package that includes both nixl-cu12 and nixl-cu13. nixl-cu13 ships a binary that breaks vllm model inspection.

I haven't figured out why the vllm-router upgrade triggered this, but the fix seems to work

Signed-off-by: SumanthRH <sumanthrh@anyscale.com>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates pyproject.toml to add an override dependency for nixl-cu13 with the environment marker sys_platform == 'never'. This change prevents the installation of nixl-cu13 on Linux, resolving a collision with nixl-cu12 that breaks vLLM model inspection in CUDA-12 environments. There are no review comments, so I have no feedback to provide.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

@SumanthRH

Copy link
Copy Markdown
Member Author

Megatron Models GPU CI passes: https://github.com/NovaSky-AI/SkyRL/actions/runs/27103746484/job/79989027090?pr=1756

The same workflow fails on main with the vllm model validation error: https://github.com/NovaSky-AI/SkyRL/actions/runs/27085057237/job/79937793420

@SumanthRH SumanthRH merged commit 689d119 into main Jun 7, 2026
9 of 11 checks passed
erictang000 added a commit that referenced this pull request Jun 8, 2026
#1756 was insufficient, since the marker on override-dependencies gates
when the override happens not when the dependency is installed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant