Requirement for Lossless Ethernet: NVMe over RoCE (RDMA over Converged Ethernet) requires a lossless fabric to function correctly. Unlike standard iSCSI which uses TCP for error recovery, RoCE assumes the network will not drop packets. If the network is "lossy," performance degrades significantly.
The Role of PFC: Priority Flow Control (PFC) (IEEE 802.1Qbb) is the specific mechanism used in Data Center Bridging (DCB) to provide flow control on a per-priority basis. It allows the switch to send a "pause" frame to the sender when buffers are full, preventing packet drops.
Symptom Analysis: In the scenario provided, the array itself is not overloaded ("within the performance envelope"). However, the addition of a new workload increased traffic to the point where buffer congestion occurred. Because PFC was likely misconfigured (either on the FlashArray ports, the network switches, or the host NICs), the network dropped packets instead of pausing traffic. This leads to "go-back-N" retransmissions and massive latency spikes that affect all workloads sharing that fabric.
Pure Storage Best Practices: Pure Storage documentation for NVMe-RoCE emphasizes that PFC must be enabled and consistent across the entire path. If there is a mismatch in PFC configuration, the resulting packet loss will cause the symptoms described: extreme latency and potential service outages.