Troubleshooting RoCEv2 PFC Deadlocks: Configuration, Compatibility & Error Resolving FAQ

Overview & Thematic Scope

In lossless Ethernet fabrics powering distributed AI training, RoCEv2 relies on Priority Flow Control to prevent packet loss—but PFC itself can create cyclic buffer dependencies that freeze the entire fabric. This troubleshooting-focused FAQ addresses the configuration missteps, ECMP path anomalies, and watchdog mechanisms that network engineers must master to eliminate PFC deadlocks in production AI clusters.

Frequently Asked Questions

Q1: What exactly is a PFC deadlock and why does it cause indefinite stalls in RoCEv2 fabrics?

A PFC deadlock is a cyclic buffer dependency where paused priority queues on multiple switches wait on each other, creating a stable loop that never resolves without intervention. In RoCEv2 lossless fabrics, this occurs when ECMP rerouting around a link failure inadvertently forms a circular path where every switch is paused, waiting for the next to drain. Once the cycle closes, the fabric buffers hold the paused frames, routing protocols believe convergence is complete, and PFC keeps reasserting on every switch—the affected lossless priority hangs indefinitely until a watchdog or manual intervention breaks the cycle.

Q2: How do I configure a PFC watchdog to automatically detect and recover from deadlocks?

Configure a per-port, per-priority PFC watchdog timer that detects continuously paused queues and automatically drains them to break the deadlock cycle. On supported platforms like Broadcom Tomahawk 4 and 5 with OcNOS-DC, the watchdog fires after a configurable interval (typically 100–400 ms) of continuous pause on a priority queue, drains the affected ingress queue by dropping frames, and restores normal PFC operation after a restore interval—all without operator intervention. The dropped frames trigger xCCL collective retransmits, which is preferable to an indefinite training job hang.

Q3: What causes PFC deadlocks to form in AI clusters with RoCEv2 traffic?

PFC deadlocks form when multi-path routing creates cyclic pause dependencies, typically triggered by ECMP hash collisions under asymmetric AI training traffic. AI clusters generate massive bursty elephant flows (GPU memory synchronization, container migration) that traditional ECMP static hashing handles poorly, causing severe hash collisions and link congestion. When congestion triggers PFC pause frames that propagate hop-by-hop, and a routing change or asymmetric load creates a circular path, the paused queues form a stable cycle—every switch waits for the next to resume, but none can because they are all paused.

Q4: How does DCQCN congestion control help prevent PFC deadlocks before they form?

DCQCN uses ECN marking and CNP feedback to reduce sender rates before ingress queues reach the PFC pause threshold, minimizing how often pause frames are asserted and reducing the window for cyclic dependencies to close. By marking packets with ECN when queue occupancy crosses a threshold, the receiver sends Congestion Notification Packets back to the sender, which then reduces its transmission rate. This end-to-end congestion control keeps queues below the PFC XOFF threshold in most cases, so pause frames are triggered far less frequently and deadlock cycles rarely get the chance to form.

Q5: What configuration parameters must be consistent across all switches to avoid triggering PFC deadlocks?

Consistent mapping of network priorities to switch traffic classes, uniform PFC enablement on lossless classes, and aligned XON/XOFF thresholds across every switch are mandatory to avoid unexpected pause behavior. If one switch has PFC enabled on priority 3 while another maps RoCEv2 traffic to priority 4, pause frames will not correctly backpressure the intended traffic, causing buffer buildup and potential deadlock formation. Similarly, mismatched ECN thresholds can cause one switch to mark congestion while another remains silent, leading to asymmetric feedback and unstable queue dynamics.

Q6: What diagnostic counters and CLI commands can I use to identify an incipient PFC deadlock?

Monitor per-priority PFC pause counters, watchdog fire events, queue occupancy trends, and gNMI telemetry streams to detect the early warning signs of deadlock formation. On Cumulus Linux, check PFC pause frame counts per interface and priority with show commands; on Cisco Nexus, the PFC watchdog statistics show queue shutdown events, drained packets, and restored events. Modern platforms stream watchdog trigger events and drain durations over gNMI, allowing SREs to treat deadlock events as actionable alerts rather than discovering silent training stalls hours later.

Q7: Why does adding more ECMP paths sometimes make PFC deadlocks worse instead of better?

More ECMP paths increase the probability that asymmetric load distribution creates circular pause dependencies, because each additional path represents another potential edge in a deadlock cycle. While ECMP is designed to distribute traffic evenly, AI training traffic is inherently bursty and elephant-flow dominated, causing hash collisions that concentrate flows on a subset of paths. When a link failure forces rerouting, the new path assignment can create a topology where switch A pauses waiting for B, B waits for C, and C waits for A—a cycle that would not exist with fewer paths or deterministic routing.

Q8: What is the relationship between PFC headroom configuration and deadlock vulnerability?

Insufficient PFC headroom causes premature pause assertion and buffer exhaustion, dramatically increasing deadlock vulnerability when routing anomalies create cyclic dependencies. Headroom is the buffer memory beyond the XOFF threshold reserved to absorb in-flight frames after a pause is sent—it must account for maximum frame size, link speed, cable length, switch and transceiver latency, and sender response time. If headroom is underestimated, packets are lost before the pause takes effect, triggering retransmissions that worsen congestion; if overestimated, precious buffer is wasted and fewer lossless priorities can be supported. Both misconfigurations can contribute to unstable pause dynamics that facilitate deadlock formation.