Near Field Notes

Videos

The notes, in motion

One video per topic, newest first. Point at a card for a silent preview.The card in the middle of the screen plays a silent preview. Open one to watch it whole.

28 videos, 28 with a note to read after.

3:14 · With sound

CRCs on fast links

How does a CRC catch bad bits, and what do fast links do when it fails?

Read Note 28

Transcript

Network switches corrupt bad frames on purpose.

A cut-through switch starts sending a frame before all of it has arrived.

The check, the CRC, comes last.

If it fails, most of the frame is already out the other port.

So the switch writes the inverse of the right CRC.

Every box after it can tell this frame was killed upstream.

Here's how CRCs work, and how fast links use them.

A CRC divides the whole frame by a fixed polynomial, using XOR instead of subtraction.

The remainder is the check. Ethernet's is 32 bits, the last 4 bytes of every frame.

In hardware, it's a shift register with XOR taps.

One bit goes in each clock. After the last bit, the register holds the remainder.

The receiver runs the frame and its check through the same register. Every good frame ends on the same constant.

A 32-bit CRC catches any burst of up to 32 flipped bits.

In a full 1518-byte frame, it catches any 3 flipped bits.

4 flipped bits slip through about once in 4 billion.

Short frames do better. Up to 375 bytes, every 4-bit error is caught.

And 9000-byte jumbo frames keep the 3-bit guarantee.

Ethernet's polynomial comes from a 1975 Air Force study of military phone lines.

The same one checks PCIe, USB 3, SATA, and SAS.

SATA starts it from its own value. In ASCII, that value reads R2P2.

Change the start value, the bit order, or the final flip, and one polynomial gives five different CRC-32s.

Smaller jobs get smaller CRCs: 5 bits on USB tokens, 8 on DDR5 writes, 16 on PCIe link packets.

What happens on a bad CRC depends on the link.

Ethernet drops the frame. TCP notices later and sends it again.

PCIe numbers every packet and keeps a copy. A bad CRC gets a NAK, and the sender replays it.

After 4 failed replays, the link retrains.

DDR memory retries too. A bad write CRC pulses an alert pin, and the controller writes again.

PCIe even has its own stomp. A switch that gives up on a packet inverts its CRC, and the spec calls that not an error.

But link CRCs can't see errors inside the boxes. In one study, up to 1 packet in 1,100 still failed TCP's check.

So storage protocols add a second CRC, end to end.

PCIe 6 moved to 4 voltage levels, and raw errors got common.

So it sends data in 256-byte flits, with error correction and an 8-byte check.

Error correction fixes what it can. The check catches what it missed, and the flit is sent again.

About 5 flits in a million get resent. That's normal now.

CXL checks the CRC first, and runs the slower correction only when the check fails. That saves about 2 ns.

Then the hardware has to keep up.

One bit per clock, a 400G port would need a 400 GHz clock.

Intel's 400G MAC takes 1024 bits per clock, at 415 MHz.

The shift register unrolls into XOR trees. Each CRC bit is one XOR of up to 554 inputs.

Only 21 come from the register itself, so it still meets timing.

At 400G, a small frame ends every 1.7 ns, so two can end in the same clock.

All that for 4 bytes at the end of a frame. And when they're wrong, the switch makes sure everyone knows.