Characterising a 200GbE RDMA Fabric on Commodity Hardware: From a Hidden PCIe x4 to PAM4 Optics

The Smoking Gun

A 200GbE RDMA fabric is supposed to be boring. Ours wasn't, and the way it failed is instructive at four different layers of the stack. A routine throughput test found one inter-node leg pinned at 58 Gbps outbound while its twin ran at 198 Gbps: a 3.4× gap that pointed away from the network and straight at the host.

The NIC had silently negotiated PCIe 4.0 x4 instead of x16. It wasn't broken. It was sitting in a chipset-connected slot, hardwired to four lanes, sharing bandwidth with the onboard secondary NICs, Wi-Fi, USB, and SATA. The link came up, the cables were fine, the switch was fine, and the card was running at 29% of its capacity. On this class of workstation board, only two of the five physical x16 slots are electrically x16 and CPU-direct; the rest hang off the chipset. The fix was geometry, not money: relocate the NIC to a CPU-direct x16 slot and flip a BIOS bifurcation setting. The leg jumped from 58 Gbps to 198 Gbps: a 3.4× improvement with zero new hardware.

The lesson generalises to any commodity multi-GPU build: a link being *up* tells you nothing about its lane width. Read the negotiated PCIe link state directly, because a starved card looks identical to a healthy one until you measure it.

The Real Ceiling: NCCL Over RoCE vs Local PCIe

With the fabric healthy, the point was to characterise what collective GPU workloads actually see. Inter-node single-direction GPU-to-GPU transfers reached 22.4–23.5 GB/s; two-rank all-reduce reached ~21–23 GB/s. For contrast, *local* intra-node GPU-to-GPU peer-to-peer over PCIe ran at ~50 GB/s, roughly 2× the inter-node ceiling. That ratio is the real subject of the exercise: a 200 Gbps RoCE fabric delivers something close to half of local PCIe peer-to-peer for collectives, which is the number that should drive parallelism decisions (and it framed an estimate that a future 400 Gbps direct-attach upgrade would land around 42–47 GB/s, with two 400G rails preferable to a single 800G link on this PCIe generation).

iperf3 verification across every leg confirmed line rate on most paths but surfaced a second, subtler ceiling: one node-to-node leg capped at ~113 Gbit/s (57% of line rate) while every host-facing path ran at 99%. That isolates the remaining bottleneck as a *host-to-host* condition (NUMA/IRQ affinity, MTU, or ring-buffer asymmetry) rather than a spine-switch fault. Naming where a ceiling *isn't* is as valuable as finding where it is.

Descending to the Physical Layer

A firmware-and-optics audit closed the loop. Firmware was already current (the OEM build actually led the public catalogue), so there was nothing to flash, but the optical diagnostics were the interesting part. The 200GBASE-SR4 link carries four lanes of 50G PAM4. Reading the optical module's per-lane diagnostics and correlating them against the link's forward-error-correction counters told a precise story:

The two weakest lanes received about 0.4–0.5 dB less light, and absorbed dramatically more corrected bits. Lane 4, getting half a dB less light than lane 2, ate roughly 150× more corrected bits (212 million vs 1.4 million). This is PAM4's signature: with four amplitude levels instead of two, a fraction of a dB of optical margin multiplies the raw bit-error rate by close to an order of magnitude.

And yet the link held a flawless full 200 Gb/s: zero link-down events, zero thermal events, a handful of thousand receive drops across hundreds of gigabytes transferred (on the order of 1e-8, operationally invisible). RS-FEC silently absorbed the entire lane asymmetry. The transmit side was uniform; the imbalance was purely receive-side optical.

Why This Matters

That last point is the real lesson, and it cuts against intuition: a link with one lane bleeding 212 million corrected bits ran perfectly, because forward error correction is doing exactly its job, masking marginal physical-layer behaviour completely. The fault is invisible at every layer above the optics. You only see it if you read the per-lane counters and line them up against optical power. On commodity hardware, fabric characterisation isn't one measurement; it's a descent from PCIe topology through NIC firmware to the optical physical layer, and a problem can hide cleanly at any one of those layers while everything above it reports green.