Spectrum-X HWPLB and xPlane: Hardware Plane Load Balancing and Failover
Summary
In a Spectrum-X multiplane network, HWPLB (Hardware Plane Load Balancing) and xPlane solve related but different problems. HWPLB inside the ConnectX SuperNIC selects the plane for each transmission decision. xPlane tracks local and remote plane availability and programs OVS-DOCA/OpenFlow forwarding so traffic can bypass a failed plane.
1. Why HWPLB is needed
A logical 800G SuperNIC may connect to several independent physical planes. Simple round-robin or static hashing keeps sending traffic to a congested, degraded, or failed plane. HWPLB hides the physical planes from the application: the OS and NCCL see one logical rail while the NIC performs hardware-based plane selection.
Application / NCCL
↓ one logical RoCE rail
ConnectX SuperNIC
├─ Plane 0
├─ Plane 1
├─ Plane 2
└─ Plane 3
2. The two-stage HWPLB decision
End-to-end congestion filtering
The SuperNIC maintains per-plane state for a destination. RTT probes, ECN/CNP feedback, rate allowance, and link state are used to remove congested or failed planes from the eligible set.
eligible_planes = all_planes - congested_planes - failed_planesLocal queue selection
Among the remaining healthy planes, the hardware compares local egress queues and favors a shallower queue. This is not simply packet_number % plane_count; it is state-aware selection.
3. What xPlane does
xPlane is the Spectrum-X/DOCA mechanism for plane availability and failover. It handles:
- Local failure: a local PF, NIC link, or host-side plane port fails;
- Remote failure: the fabric reports that a remote route or plane path is unavailable.
xPlane propagates this state to OVS-DOCA and updates forwarding rules so traffic is diverted away from the failed plane. It does not replace HWPLB: it provides the availability signal that defines which planes are usable.
4. How HWPLB and xPlane work together
Plane link flap or fabric route failure
↓
xPlane detects or receives the failure
↓
OVS-DOCA forwarding rules are updated
↓
The failed plane is removed from usable paths
↓
HWPLB continues selecting among healthy, shallow-queue planes
↓
RoCE/NCCL continues to use the logical railHWPLB is the high-speed NIC data-plane selector. xPlane manages plane health, failover state, and host-side forwarding. In simple terms: HWPLB decides “which healthy path,” while xPlane decides “whether this path is currently usable.”
5. Reordering, RoCE, and PCIe
Different planes can have different queueing and path latency, so dynamic selection may create packet reordering. ConnectX high-speed packet reordering uses RoCE transport context, QP state, and sequence information such as PSN to identify gaps, hold or track packets that arrive early, and release data in the required order for RDMA placement and completions.
PCIe Gen6 provides high-bandwidth DMA between the NIC and GPU/CPU. It does not select network planes and it does not perform RoCE packet ordering.
6. What happens when one plane fails?
With four equally sized planes, the ideal remaining capacity after a permanent single-plane failure is approximately 75%; with two planes it is approximately 50%. Actual NCCL P99 and throughput depend on detection and failover time, retransmissions, DCQCN recovery, PFC/ECN behavior, collective stragglers, and topology balance. These percentages are capacity estimates, not universal SLAs.
A failure test should capture timestamps, xPlane state transitions, local/remote failures, diverted traffic, NCCL P99/P999, AllReduce throughput, PFC/ECN/CNP, retransmissions, and FEC errors.
7. Production monitoring
kubectl -n nvidia-network-operator get ds xplane-ds
kubectl -n nvidia-network-operator logs -l app=xplane --since=10m
kubectl -n nvidia-network-operator exec ds/xplane-ds -- doca-xplane-client get-status
kubectl -n nvidia-network-operator exec ds/xplane-ds -- doca-xplane-client get-planes-summary
Export stable Prometheus metrics such as:
spectrum_xplane_overall_state{cluster,node,rail,state}
spectrum_xplane_plane_state{cluster,node,rail,plane,state}
spectrum_xplane_failure_total{cluster,node,rail,plane,type}
spectrum_xplane_diverted_bytes_total{cluster,node,rail,plane}
spectrum_xplane_exporter_up{cluster,node}Alert on sustained DEGRADED/FAILED/UNKNOWN overall state, a FAILED plane, repeated flaps, continuously growing diverted traffic, and exporter unavailability. UNKNOWN must not be treated as HEALTHY.
Conclusion
HWPLB selects planes, xPlane manages plane health and failover, RoCE reordering preserves transport order, and PFC/ECN/DCQCN manage congestion. Production acceptance must prove that traffic really moves away from a failed plane, healthy planes continue to carry traffic, NCCL degradation matches the measured baseline, and recovery does not cause configuration flapping.