Uneven GPU utilization in a multi-GPU setup often stems from an imbalanced data loading process. In distributed training, if data isn't evenly distributed across GPUs (e.g., via data parallelism), some GPUs receive more work while others idle, causing performance slowdowns. NVIDIA's NCCL ensures efficient communication between GPUs, but it relies on the data pipeline-managed by tools like NVIDIA DALI or PyTorch DataLoader-to distribute batches uniformly. A bottleneck in data loading, such as slow I/O or poor partitioning, is a common culprit, detectable via NVIDIA profiling tools like Nsight Systems.
Model code optimized for specific GPUs (Option A) is unlikely unless explicitly written to exclude certain GPUs, which is rare. Different GPU models (Option B) can cause imbalances due to varying capabilities, but NVIDIA frameworks typically handle heterogeneity; this would be a design flaw, not a sudden issue.
Improper installation (Option C) would likely cause complete failures, not partial utilization. Data distribution is the most probable and fixable cause, per NVIDIA's distributed training best practices.