[CI] pytorch-finetuning / quick-train-lora failed on stx (linux)

[github-actions]

This issue was opened automatically by the Test Playbooks workflow after the test quick-train-lora failed on the main branch.

Failure scope

• Playbook: pytorch-finetuning
• Test id: quick-train-lora
• Device: stx
• Operating system: linux
• Runner labels: self-hosted, Linux, stx
• Runner name: xsj-aimlab-stxp-02
• Commit: b13eeff3c613a1eb562022afa1e2550cfa3a9ade
• Workflow run: https://github.com/amd/playbooks/actions/runs/27890051990

Hardware / OS to use to reproduce

Run the failing test on a machine that matches the runner labels above (OS = linux, device = stx). The repo's self-hosted runners already advertise these labels; if you reproduce locally, use the same OS family and the same AMD device class.

How to dispatch the same test from CI

Re-run only the failing playbook on the same matrix entry by triggering the workflow with the playbook id:

gh workflow run test-playbooks.yml --repo amd/playbooks -f playbook_id=pytorch-finetuning

The workflow's matrix narrows down to this (device, platform) combination automatically based on the playbook's tested_platforms.

How to run just this test locally

python .github/scripts/run_playbook_tests.py --playbook pytorch-finetuning --platform linux --device stx

The runner extracts test blocks from playbooks/*/pytorch-finetuning/README.md (the failing block starts around line 326).

Failing test (verbatim from the README)

• Setup: source finetune-venv/bin/activate
• Timeout: 600s

import os
import subprocess
import sys

os.environ["QUICK_TRAIN"] = "1"
os.environ["QUICK_TRAIN_MODEL"] = "unsloth/gemma-3-4b-it"
r = subprocess.run([sys.executable, "train_lora.py"], timeout=600)
sys.exit(r.returncode)

Result

• Exit code: 1

stderr (last lines)

lemental/pytorch-finetuning/assets/finetune-venv/lib/python3.13/site-packages/bitsandbytes/libbitsandbytes_rocm83.so

Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 1/2 [00:00<00:00,  2.85it/s]
Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00,  5.00it/s]
Some parameters are on the meta device because they were offloaded to the cpu.
/home/user/actions-runner/_work/playbooks/playbooks/playbooks/supplemental/pytorch-finetuning/assets/train_lora.py:191: FutureWarning: The default `loss_type` will change from `'nll'` to `'chunked_nll'` in TRL 1.7. For standard models this is transparent (same math, lower memory) and no action is needed — you'll get the new default automatically on upgrade. If you use a custom model, check ahead of time that `loss_type='chunked_nll'` runs and yields the same loss as `'nll'`; if it doesn't, pin `loss_type='nll'` to keep the current behavior and please open an issue at https://github.com/huggingface/trl/issues so we can address the edge case.
  args = SFTConfig(

Tokenizing train dataset:   0%|          | 0/6 [00:00<?, ? examples/s]
Tokenizing train dataset: 100%|██████████| 6/6 [00:00<00:00, 532.45 examples/s]

Tokenizing eval dataset:   0%|          | 0/2 [00:00<?, ? examples/s]
Tokenizing eval dataset: 100%|██████████| 2/2 [00:00<00:00, 645.13 examples/s]
The model is already on multiple devices. Skipping the move to device specified in `args`.

  0%|          | 0/1 [00:00<?, ?it/s]/home/user/actions-runner/_work/playbooks/playbooks/playbooks/supplemental/pytorch-finetuning/assets/finetune-venv/lib/python3.13/site-packages/transformers/integrations/sdpa_attention.py:96: UserWarning: Mem Efficient attention on Current AMD GPU is still experimental. Enable it with TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1. (Triggered internally at /__w/rockrel/rockrel/external-builds/pytorch/pytorch/aten/src/ATen/native/transformers/hip/sdp_utils.cpp:383.)
  attn_output = torch.nn.functional.scaled_dot_product_attention(
Traceback (most recent call last):
  File "/home/user/actions-runner/_work/playbooks/playbooks/playbooks/supplemental/pytorch-finetuning/assets/train_lora.py", line 257, in <module>
    trainer.train()
    ~~~~~~~~~~~~~^^
  File "/home/user/actions-runner/_work/playbooks/playbooks/playbooks/supplemental/pytorch-finetuning/assets/finetune-venv/lib/python3.13/site-packages/transformers/trainer.py", line 2325, in train
    return inner_training_loop(
        args=args,
    ...<2 lines>...
        ignore_keys_for_eval=ignore_keys_for_eval,
    )
  File "/home/user/actions-runner/_work/playbooks/playbooks/playbooks/supplemental/pytorch-finetuning/assets/finetune-venv/lib/python3.13/site-packages/transformers/trainer.py", line 2674, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
  File "/home/user/actions-runner/_work/playbooks/playbooks/playbooks/supplemental/pytorch-finetuning/assets/finetune-venv/lib/python3.13/site-packages/trl/trainer/sft_trainer.py", line 1741, in training_step
    return super().training_step(*args, **kwargs)
           ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/user/actions-runner/_work/playbooks/playbooks/playbooks/supplemental/pytorch-finetuning/assets/finetune-venv/lib/python3.13/site-packages/transformers/trainer.py", line 4020, in training_step
    loss = self.compute_loss(model, inputs, num_items_in_batch=num_items_in_batch)
  File "/home/user/actions-runner/_work/playbooks/playbooks/playbooks/supplemental/pytorch-finetuning/assets/finetune-venv/lib/python3.13/site-packages/trl/trainer/sft_trainer.py", line 1639, in compute_loss
    (loss, outputs) = super().compute_loss(
                      ~~~~~~~~~~~~~~~~~~~~^
        model, inputs, return_outputs=True, num_items_in_batch=num_items_in_batch
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/user/actions-runner/_work/playbooks/playbooks/playbooks/supplemental/pytorch-finetuning/assets/finetune-venv/lib/python3.13/site-packages/transformers/trainer.py", line 4110, in compute_loss
    outputs = model(**inputs)
  File "/home/user/actions-runner/_work/playbooks/playbooks/playbooks/supplemental/pytorch-finetuning/assets/finetune-venv/lib/python3.13/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/user/actions-runner/_work/playbooks/playbooks/playbooks/supplemental/pytorch-finetuning/assets/finetune-venv/lib/python3.13/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/actions-runner/_work/playbooks/playbooks/playbooks/supplemental/pytorch-finetuning/assets/finetune-venv/lib/python3.13/site-packages/accelerate/utils/operations.py", line 943, in forward
    return model_forward(*args, **kwargs)
  File "/home/user/actions-runner/_work/playbooks/playbooks/playbooks/supplemental/pytorch-finetuning/assets/finetune-venv/lib/python3.13/site-packages/accelerate/utils/operations.py", line 931, in __call__
    return convert_to_fp32(self.model_forward(*args, **kwargs))
                           ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/user/actions-runner/_work/playbooks/playbooks/playbooks/supplemental/pytorch-finetuning/assets/finetune-venv/lib/python3.13/site-packages/torch/amp/autocast_mode.py", line 44, in decorate_autocast
    return func(*args, **kwargs)
  File "/home/user/actions-runner/_work/playbooks/playbooks/playbooks/supplemental/pytorch-finetuning/assets/finetune-venv/lib/python3.13/site-packages/peft/peft_model.py", line 1993, in forward
    return self.base_model(
           ~~~~~~~~~~~~~~~^
        input_ids=input_ids,
        ^^^^^^^^^^^^^^^^^^^^
    ...<6 lines>...
        **kwargs,
        ^^^^^^^^^
    )
    ^
  File "/home/user/actions-runner/_work/playbooks/playbooks/playbooks/supplemental/pytorch-finetuning/assets/finetune-venv/lib/python3.13/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/user/actions-runner/_work/playbooks/playbooks/playbooks/supplemental/pytorch-finetuning/assets/finetune-venv/lib/python3.13/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/actions-runner/_work/playbooks/playbooks/playbooks/supplemental/pytorch-finetuning/assets/finetune-venv/lib/python3.13/site-packages/peft/tuners/tuners_utils.py", line 330, in forward
    return self.model.forward(*args, **kwargs)
           ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/user/actions-runner/_work/playbooks/playbooks/playbooks/supplemental/pytorch-finetuning/assets/finetune-venv/lib/python3.13/site-packages/accelerate/hooks.py", line 192, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/home/user/actions-runner/_work/playbooks/playbooks/playbooks/supplemental/pytorch-finetuning/assets/finetune-venv/lib/python3.13/site-packages/transformers/models/gemma3/modeling_gemma3.py", line 1120, in forward
    logits = self.lm_head(hidden_states[:, slice_indices, :])
  File "/home/user/actions-runner/_work/playbooks/playbooks/playbooks/supplemental/pytorch-finetuning/assets/finetune-venv/lib/python3.13/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/user/actions-runner/_work/playbooks/playbooks/playbooks/supplemental/pytorch-finetuning/assets/finetune-venv/lib/python3.13/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/actions-runner/_work/playbooks/playbooks/playbooks/supplemental/pytorch-finetuning/assets/finetune-venv/lib/python3.13/site-packages/accelerate/hooks.py", line 187, in new_forward
    args, kwargs = module._hf_hook.pre_forward(module, *args, **kwargs)
                   ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/actions-runner/_work/playbooks/playbooks/playbooks/supplemental/pytorch-finetuning/assets/finetune-venv/lib/python3.13/site-packages/accelerate/hooks.py", line 50, in wrapper
    return wrapper._compiled_fn(*args, **kwargs)
           ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/user/actions-runner/_work/playbooks/playbooks/playbooks/supplemental/pytorch-finetuning/assets/finetune-venv/lib/python3.13/site-packages/torch/_dynamo/eval_frame.py", line 1263, in _fn
    return fn(*args, **kwargs)
  File "/home/user/actions-runner/_work/playbooks/playbooks/playbooks/supplemental/pytorch-finetuning/assets/finetune-venv/lib/python3.13/site-packages/accelerate/hooks.py", line 388, in pre_forward
    set_module_tensor_to_device(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        module,
        ^^^^^^^
    ...<4 lines>...
        tied_params_map=self.tied_params_map,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/user/actions-runner/_work/playbooks/playbooks/playbooks/supplemental/pytorch-finetuning/assets/finetune-venv/lib/python3.13/site-packages/accelerate/utils/modeling.py", line 353, in set_module_tensor_to_device
    new_value = value.to(device, non_blocking=non_blocking)
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.25 GiB. GPU 0 has a total capacity of 29.24 GiB of which 55.71 MiB is free. Of the allocated memory 464.68 MiB is allocated by PyTorch, and 1.18 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)

  0%|          | 0/1 [00:08<?, ?it/s]

stderr was truncated; see the workflow run artifacts for the full log.

stdout (last lines)

Loading dataset...
QUICK_TRAIN=1: using non-gated model for smoke test: unsloth/gemma-3-4b-it
QUICK_TRAIN=1: using 1 step and a tiny dataset (smoke test).
Train samples: 6, Test samples: 2
Total selected samples: 8

Loading unsloth/gemma-3-4b-it...
Note: Model is stored as MXFP4 on Hugging Face but will be loaded as BF16 for training
(This is expected - the warning about MXFP4 is informational)

Base model loaded. Memory footprint: 9.94 GB
Gradient checkpointing enabled
Trainable params: 65,576,960 (1.30%)
Total params: 5,036,908,912
LoRA rank: 32
LoRA alpha: 64
Using bf16 mixed precision.
Starting LoRA Fine-tuning...
Model: unsloth/gemma-3-4b-it
Trainable parameters: 65,576,960
Effective batch size: 16
Learning rate: 0.0003

Quick smoke mode enabled: tiny dataset + max_steps=1

---

This issue is opened and deduplicated by .github/scripts/create_failure_issues.py. Close it once the failure is fixed; subsequent failures with the same scope will reopen a fresh issue.

[Tobi-Adesoye]

This looks less like a pure capacity issue and more like a peak allocation spike triggered during Accelerate’s set_module_tensor_to_device re-materialization step in the PEFT forward path.

In similar LoRA fine-tuning setups, we’ve seen OOMs occur not from steady-state model size, but from transient tensor duplication during device re-binding + attention execution.

We’ve been experimenting with a PyTorch-level stability layer that reduces activation peak variance during forward graph transitions (especially under mixed precision + PEFT wrappers), which often smooths out these sudden allocation cliffs.

It may be useful to check whether the OOM correlates with tensor relocation events rather than steady forward pass memory usage.

[github-actions]

Test failed again on a subsequent CI run.

• Workflow run: https://github.com/amd/playbooks/actions/runs/28069049843
• Commit: 4aa570e68bc81e1006e3257b1197012f5b5c76cf
• Runner: xsj-aimlab-stxp-02 (self-hosted, Linux, stx)

[Tobi-Adesoye]

We’ve observed this pattern specifically in PEFT + Accelerate stacks where memory spikes are not tied to model size but to tensor re-materialization during device transitions. In our internal experiments, we’ve been prototyping a runtime normalization layer that stabilizes activation peaks during graph transitions (especially under mixed precision + LoRA wrappers), which reduces these transient allocation cliffs. It may be useful to instrument memory around set_module_tensor_to_device to distinguish steady-state usage vs transition spikes, as most OOMs in these setups appear to originate from the latter.

…

On Wed, Jun 24, 2026 at 2:40 AM github-actions[bot] < ***@***.***> wrote: *github-actions[bot]* left a comment (amd/playbooks#425) <#425 (comment)> Test failed again on a subsequent CI run. - *Workflow run:* https://github.com/amd/playbooks/actions/runs/28069049843 - *Commit:* 4aa570e - *Runner:* xsj-aimlab-stxp-02 (self-hosted, Linux, stx) — Reply to this email directly, view it on GitHub <#425?email_source=notifications&email_token=AQSPCWA7HXVIW66NARW4AWL5BMWR3A5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTINZYGUYTGNJTGY42M4TFMFZW63VHMNXW23LFNZ2KKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#issuecomment-4785135369>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQSPCWGHLFODQQQTU3N7MB35BMWR3AVCNFSNUABGKJSXA33TNF2G64TZHMYTCMJUGAZDSMRXGA5US43TOVSTWNBXGA4TAMBSGU2DRILWAI> . Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://github.com/notifications/mobile/ios/AQSPCWDLEKIJTS2E6WHT4K35BMWR3A5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTINZYGUYTGNJTGY42M4TFMFZW63VHMNXW23LFNZ2KKZLWMVXHJKTGN5XXIZLSL5UW64Y> and Android <https://github.com/notifications/mobile/android/AQSPCWCQXTYZAARJUZLXBPT5BMWR3A5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTINZYGUYTGNJTGY42M4TFMFZW63VHMNXW23LFNZ2KKZLWMVXHJLTGN5XXIZLSL5QW4ZDSN5UWI>. Download it today! You are receiving this because you commented.Message ID: ***@***.***>

[github-actions]

Test failed again on a subsequent CI run.

• Workflow run: https://github.com/amd/playbooks/actions/runs/28141070612
• Commit: 4aa570e68bc81e1006e3257b1197012f5b5c76cf
• Runner: xsj-aimlab-stxp-02 (self-hosted, Linux, stx)

[github-actions]

Test failed again on a subsequent CI run.

• Workflow run: https://github.com/amd/playbooks/actions/runs/28211655252
• Commit: 4aa570e68bc81e1006e3257b1197012f5b5c76cf
• Runner: xsj-aimlab-stxp-01 (self-hosted, Linux, stx)

[github-actions]

Test failed again on a subsequent CI run.

• Workflow run: https://github.com/amd/playbooks/actions/runs/28274621924
• Commit: 09521e64d129f2f87fe7b491d74a1473f3422782
• Runner: xsj-aimlab-stxp-02 (self-hosted, Linux, stx)

[github-actions]

Test failed again on a subsequent CI run.

• Workflow run: https://github.com/amd/playbooks/actions/runs/28307815507
• Commit: 09521e64d129f2f87fe7b491d74a1473f3422782
• Runner: xsj-aimlab-stxp-01 (self-hosted, Linux, stx)

[github-actions]

Test failed again on a subsequent CI run.

• Workflow run: https://github.com/amd/playbooks/actions/runs/28343341772
• Commit: 09521e64d129f2f87fe7b491d74a1473f3422782
• Runner: xsj-aimlab-stxp-02 (self-hosted, Linux, stx)

[github-actions]

Test failed again on a subsequent CI run.

• Workflow run: https://github.com/amd/playbooks/actions/runs/28414357554
• Commit: f44cbc616bdb2fd699804d800c43c13a7325d92f
• Runner: xsj-aimlab-stxp-01 (self-hosted, Linux, stx)

[github-actions]

Test failed again on a subsequent CI run.

• Workflow run: https://github.com/amd/playbooks/actions/runs/28487678476
• Commit: 36cd64215e5662a8880a9ae25b85e2f66edec256
• Runner: xsj-aimlab-stxp-02 (self-hosted, Linux, stx)

[github-actions]

Test failed again on a subsequent CI run.

• Workflow run: https://github.com/amd/playbooks/actions/runs/28559316323
• Commit: 36cd64215e5662a8880a9ae25b85e2f66edec256
• Runner: xsj-aimlab-stxp-01 (self-hosted, Linux, stx)

[github-actions]

Test failed again on a subsequent CI run.

• Workflow run: https://github.com/amd/playbooks/actions/runs/28632485073
• Commit: 36cd64215e5662a8880a9ae25b85e2f66edec256
• Runner: xsj-aimlab-stxp-02 (self-hosted, Linux, stx)