While fine-tuning the network (on 4 V100 GPUs) for text to SC, I always face the issue of CUDA OUT OF MEMORY.
If the batch size is reduced to 2 (from 16), this issue is faced again.
I am currently using -
pytorch 1.10.2
transformers 3.4.0
wandb==0.12.16
huggingface-hub==0.6.0
batch_size: 2
num_workers: 8
E.g.:
RuntimeError: CUDA out of memory. Tried to allocate 194.00 MiB (GPU 5; 23.59 GiB total capacity; 22.18 GiB already allocated;111.19 MiB free; 23.37 GiB reserved in total by PyTorch).
Did you face any of these issues during fine-tuning?
While fine-tuning the network (on 4 V100 GPUs) for text to SC, I always face the issue of CUDA OUT OF MEMORY.
If the batch size is reduced to 2 (from 16), this issue is faced again.
I am currently using -
pytorch 1.10.2
transformers 3.4.0
wandb==0.12.16
huggingface-hub==0.6.0
batch_size: 2
num_workers: 8
E.g.:
RuntimeError: CUDA out of memory. Tried to allocate 194.00 MiB (GPU 5; 23.59 GiB total capacity; 22.18 GiB already allocated;111.19 MiB free; 23.37 GiB reserved in total by PyTorch).
Did you face any of these issues during fine-tuning?