fix(recipes): remove deprecated fully_shard kwargs (check_for_nan_in_grad, grad_reduce_in_fp32)

[svc-bionemo]

Summary

Fixes nightly CI failures caused by updated megatron-fsdp package that removed the check_for_nan_in_grad and grad_reduce_in_fp32 keyword arguments from fully_shard().

Failures Fixed

• recipes/geneformer_native_te_mfsdp_fp8 (3 tests): TypeError: fully_shard() got an unexpected keyword argument 'check_for_nan_in_grad'
• recipes/esm2_native_te (8 tests): TypeError: fully_shard() got an unexpected keyword argument 'check_for_nan_in_grad'
• recipes/vit (3 tests): TypeError: fully_shard() got an unexpected keyword argument 'grad_reduce_in_fp32'

Changes

• Removed check_for_nan_in_grad and grad_reduce_in_fp32 from fully_shard_kwargs in geneformer and esm2 hydra configs
• Removed grad_reduce_in_fp32 from vit config files and train.py

Not Fixed (infrastructure issue)

• models/geneformer (4 tests): AssertionError: Minimum required nvidia-resiliency-ext package version is 0.6.0. — This is a container image issue requiring nvidia-resiliency-ext>=0.6.0 to be installed.

CI Run

https://github.com/NVIDIA-BioNeMo/bionemo-framework/actions/runs/26814311142

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 1b67102f-029c-4a0e-b6b1-c3c07f289261

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[pstjohn]

closing in favor of #1585