Bypassing the Kernel: How RDMA Made GPU Clusters Fast Enough for AI

"The network is the computer." John Gage coined that line for Sun Microsystems in 1984, at the time a pitch for distributed computing that most of the industry filed under marketing. Four decades later, frontier AI training and inference made it literal: a training cluster is thousands of GPUs that only behave like one computer if the network between them keeps up. This is the first post in a series on how that happened, starting with the biggest shift of all: AI clusters gave up on the kernel's networking stack entirely.

Here's what "keeps up" means in practice. An H100 finishes a matrix multiply in microseconds. Getting that result to the GPU next door (across a switch, through the kernel, into another process's memory) can take orders of magnitude longer: a plain TCP socket round-trip commonly costs somewhere in the 10–50 microsecond range once you count the kernel copy, the context switch, and the interrupt, while RDMA does the same job in roughly 1–2 microseconds. In a training run spanning thousands of GPUs, the profiler often points not at compute kernels but at the network stack itself.

The fix wasn't a faster GPU. It was skipping the kernel entirely.

To see how, it helps to look first at what the standard networking stack has been doing, largely unchanged, for the last thirty years.

The old default: TCP/IP sockets

Every general-purpose data center still runs on the same basic model: an application opens a socket, hands data to the kernel, and the kernel takes it from there. The kernel copies that data into its own buffers, runs it through the TCP/IP stack, hands it to a NIC driver, and the driver puts it on the wire. On the receiving side, the path runs in reverse: the NIC raises an interrupt, the kernel copies the incoming data into its buffers, and eventually into the application's memory.

This works, and it's why sockets are the default for essentially everything. But every hop in that path costs something: a buffer copy, a context switch between user space and kernel space, an interrupt that has to be serviced. None of it is expensive in isolation. At the scale of a single request, it's noise.

It stops being noise when thousands of GPUs are synchronizing on every training step, each one exchanging gradients with every other one, and, inside tensor- and pipeline-parallel training, running many smaller collective operations per second on top of that. At that point, the same per-packet overhead that's invisible in a web request becomes the dominant cost in the system, and not just in wall-clock time: idle GPU-hours on a cluster renting for several dollars per GPU-hour land directly on the training bill, not just in a benchmark footnote.

Here's what that path actually looks like, hop by hop:

Enter RDMA: the kernel gets cut out

RDMA (Remote Direct Memory Access) starts from a different premise: let the network card, concretely something like an NVIDIA ConnectX-7 or BlueField-3, read and write application memory directly, without the kernel touching the data and without waking up the remote CPU at all. The mechanism that makes this possible is worth walking through carefully, because "kernel bypass" isn't one trick. It's a small set of structures working together: memory regions, queue pairs, and completion queues.

Memory regions. Before any transfer, both sides register a chunk of application memory with the RDMA NIC (the RNIC) by calling ibv_reg_mr(). The kernel pins those pages so they can't be swapped out, and the RNIC builds address-translation tables for them. Registration hands back two keys: an lkey, used by the local RNIC to touch the buffer, and an rkey, which gets shared with the remote peer so its RNIC is allowed to target this memory too. This one-time registration, along with the queue-pair creation and state transitions described next, is the only place the kernel gets involved. Once the queue pair is live, the data path never calls into it again.

Queue pairs. Every RDMA connection is a queue pair: a send queue and a receive queue, created once via ibv_create_qp() and moved through a small state machine (INIT → RTR → RTS) via ibv_modify_qp(). Once the queue pair is up, the application talks to it directly from user space. There's no socket, no syscall per message, just a data structure the app and the RNIC both have access to.

Completion queues. When a queue pair is created, it's associated with a completion queue (CQ). This is where the RNIC reports that work finished, and it's how the application learns "done" without being interrupted along the way.

With those three pieces in place, sending data looks like this:

1. The app builds a work request (WQE), carrying a source address and lkey, a destination address and rkey, and a length, and hands it to the queue pair with ibv_post_send().
2. It rings the RNIC's doorbell: a single memory-mapped I/O (MMIO) write telling the hardware a new WQE is ready. This is the kernel-bypass moment. No system call, no context switch, on the fast path.
3. The RNIC's DMA engine reads the payload straight out of the registered memory region, validated against the lkey. This is zero-copy: the data never touches a kernel buffer or a bounce buffer.
4. The RNIC packetizes the data with RDMA transport headers (a BTH carrying the queue-pair number, opcode, and packet sequence number, plus a RETH carrying the remote address, rkey, and length) and sends it across the fabric.
5. The remote RNIC validates the rkey, translates the remote virtual address, and DMA-writes the payload directly into the target's memory region, without interrupting the remote CPU or the remote application at all.
6. On a reliable-connected transport, the remote RNIC sends an ACK back. The initiator's RNIC posts a completion queue entry (CQE), and the application discovers it by calling ibv_poll_cq().

That's an RDMA_WRITE: a one-sided operation, where the initiator names both the local and remote address, and the target's CPU is never involved. RDMA_READ works the same way in reverse. Both differ from a third mode, SEND/RECV, which is two-sided: the receiver has to pre-post a RECV work request naming where incoming data is allowed to land, and, unlike WRITE/READ, the receiving application does get a completion event, because it needs to know a message arrived. WRITE and READ are how RDMA gets its "the remote CPU was never touched" property; SEND/RECV trades that property for something closer to a traditional message-passing model, still without the kernel or a copy.

Here's the same sequence, animated. Pick WRITE, READ, or SEND and step through it:

What this looks like in code

The concepts above map onto a real API: libibverbs, the userspace verbs library that sits directly on top of the RNIC driver. Setup happens once, outside the hot path:

/* one-time setup: register memory, then create + bring up a queue pair */
struct ibv_mr *mr = ibv_reg_mr(pd, buf, buf_size,
    IBV_ACCESS_LOCAL_WRITE | IBV_ACCESS_REMOTE_WRITE | IBV_ACCESS_REMOTE_READ);

struct ibv_qp *qp = ibv_create_qp(pd, &qp_init_attr);
/* qp then moves INIT -> RTR -> RTS via ibv_modify_qp(), exchanging
   the remote's QP number, rkey, and buffer address over an
   out-of-band channel (plain TCP, or librdmacm) before the fast
   path below ever runs */

That last comment matters: RDMA still needs a slow path to bootstrap the fast one. The peers have to agree on queue-pair numbers, keys, and addresses somehow, and that handshake typically happens over an ordinary TCP connection (or librdmacm, which wraps the same idea). RDMA replaces the data path, not the very first handshake.

Once the queue pair is live, the part that runs on every message is small:

/* hot path: post an RDMA WRITE, then poll for its completion */
struct ibv_sge sge = {
    .addr   = (uintptr_t)buf,
    .length = buf_size,
    .lkey   = mr->lkey,
};

struct ibv_send_wr wr = {0}, *bad_wr;
wr.wr_id               = 1;
wr.sg_list             = &sge;
wr.num_sge             = 1;
wr.opcode              = IBV_WR_RDMA_WRITE;
wr.send_flags          = IBV_SEND_SIGNALED;
wr.wr.rdma.remote_addr = remote_addr;  /* learned during setup, above */
wr.wr.rdma.rkey        = remote_rkey;  /* learned during setup, above */

ibv_post_send(qp, &wr, &bad_wr);   /* rings the doorbell; no syscall here */

struct ibv_wc wc;
while (ibv_poll_cq(cq, 1, &wc) == 0) {
    /* spin, or block on a completion channel instead */
}
/* omitted for brevity: ibv_poll_cq() itself returns <0 on error,
   which a production version needs to check separately from wc.status */
if (wc.status != IBV_WC_SUCCESS) {
    /* handle error */
}

ibv_post_send() is the doorbell ring from step 2 above. ibv_poll_cq() is the app reaping the CQE from step 6. Everything in between (the DMA read, the packetization, the remote DMA write, the ACK) happens in the RNIC and on the wire, with neither CPU making a system call.

RDMA vs TCP/IP, side by side

The setup row matters: RDMA isn't strictly "better." It trades a cheap, flexible connection model for a more expensive one-time setup, a trade that only pays off once traffic is frequent enough to amortize the fixed cost. That's exactly the shape of GPU collective communication during training.

Why AI training specifically

None of this is new. RDMA has existed in HPC and financial trading for two decades. What changed is that AI training produces the traffic pattern RDMA was built for: a fixed set of peers (the GPUs in a job), exchanging large buffers (gradients, activations), over and over, for the entire duration of a run. Two pieces of AI infrastructure lean on exactly the mechanism described above:

• GPUDirect RDMA extends the same idea one hop further: the RNIC's DMA engine reads and writes GPU memory directly, so a gradient never has to be staged through host RAM at all on its way to or from another GPU.
• NCCL, the collective-communications library nearly all distributed training frameworks sit on top of, uses RDMA transports under the hood for its all-reduce, all-gather, and broadcast operations whenever the underlying network supports it, falling back to TCP sockets only when it doesn't.

Closing

Amdahl's Law says a system's overall speedup is capped by whatever part of it doesn't get faster, no matter how much you parallelize everything else. Scale a training job from one GPU to ten thousand, and the network between them is exactly that part: however fast each GPU computes, the job as a whole moves at the speed of the slowest hop between two of them.

Gage's line was about distributed computing in general. AI training is the case where it stopped being a metaphor: a cluster's GPUs only compute as one machine if the network stops acting like a general-purpose network and starts acting like an extension of memory. That's what RDMA is. And it's why, a decade after most of the industry filed it under "niche HPC technology," it's the default.

Next in this series: this post focused on the mechanism, why kernel bypass exists and what it buys you. It deliberately left out how RDMA actually gets deployed, which transport carries it (InfiniBand vs. RoCE vs. iWARP), how a lossless-enough Ethernet fabric gets built for RoCE using DCB and priority flow control, and how cloud providers expose RDMA NICs to individual VMs via SR-IOV. That's Three Wires, One Promise.