Inmulti-subnet InfiniBand environments, AI training clusters are segmented across network zones (or subnets). Direct GPU-to-GPU communication (especially for collective ops like AllReduce, Broadcast, etc.) requires inter-subnet reachability. NCCL supports this via theInfiniBand Router (IB Router)feature.
From theNCCL User Guide - Environment Variables Section:
"NCCL_IB_USE_IB_ROUTER: Enables NCCL support for IB routers which are used in multi-subnet InfiniBand fabrics. When enabled, NCCL can traverse IB subnets using a properly configured IB router." This is critical because without IB Router support:
* NCCL would be restricted to intra-subnet GPU collectives.
* Multi-node training across subnets would fail or fall back to slower TCP fallback mechanisms.
Technical Explanation:
* IB Routers usesubnet managers(like OpenSM with routing tables) to bridge communication across different InfiniBand partitions.
* NCCL queries the subnet topology, discovers routing paths, and usesRDMA CM(Connection Manager) to establish GPU transport over routers.
* This capability is especially important in data center-scale AI clusters spanning multiple racks or zones, connected viaIB routers like Mellanox SB7800 or QM8700 series.
When NCCL_IB_USE_IB_ROUTER=1 is set:
* NCCL includes router-aware route resolution in its path selection logic.
* Enables efficientzero-copy communicationacross GPUs in different IB domains, maintaining low latency.
Other Options Explained:
* A. Lazy connection establishment- controls when peer connections are made but does not enable cross-subnet reach.
* B. GPU Direct RDMA- enables intra-node direct memory access, not applicable for routing across subnets.
* C. Static plugin linking- affects how NCCL links plugins, not related to IB topology.
Exact Extract Reference:
Source: NVIDIA NCCL User Guide - Environment Variables Section
Extract: "NCCL_IB_USE_IB_ROUTER: Enables NCCL support for IB routers, required for multi-subnet InfiniBand configurations. Ensures proper routing of collectives over fabric-wide topologies."