Error correction on fast links
How big an error can a fast link fix by itself?
From one wrong bit in a camera packet’s header to 141 wrong bits in a row on 400G Ethernet, wherever they land. That is why a 400G link can get 102 million bits wrong a second and stay in spec. At that limit a block carries 1.30 bad symbols on average against the 15 it can fix, and one block is lost every 4.36 hours.
The receiver finds and fixes the wrong bits from check symbols the sender adds, with nothing sent again.
The margin ends in a cliff. Four times the errors loses 119 million times more blocks.
Fast links are allowed to get bits wrong
102 million a secondPCIe up to version 5 was built to get almost every bit right: one wrong bit in 10¹². On a 32 Gb/s lane that is one every 31 seconds.
PCIe 6 and 400G Ethernet send four voltage levels instead of two, and their specs allow far more. PCIe 6 allows a first-bit error rate of 10⁻⁶. 400G Ethernet’s optical link allows 2.4 wrong bits in 10,000.
A 400G module like this one sends 425 Gb/s on the line, eight lanes of 53.125 Gb/s. At 2.4 × 10⁻⁴ that is 102 million wrong bits every second, inside the spec.
Counted a second, a PCIe 5 lane was allowed one wrong bit every 31 seconds and a 400G link is allowed 102 million. Each spec counts in its own way, listed under the chart.
With no correction, 94.6% of full-size 1,518-byte Ethernet frames would arrive damaged at that rate, and 11.6% of the smallest, 64 bytes.
So the sender adds check symbols worked out from the data, and the receiver uses them to find the wrong bits and put them right. This is forward error correction. It started with one bit.
The numbers
Each spec counts its own way. PCIe 6 counts a burst once, by its first wrong bit.
| Link | Errors a second | What the spec counts |
|---|---|---|
| PCIe 5, one lane | 1 every 31 s | wrong bits, 10⁻¹² at 32 Gb/s |
| Thunderbolt 5, one lane | 256 | wrong symbols, 10⁻⁸ at 25.6 GBd |
| PCIe 6, one lane | 64,000 | first wrong bits, 10⁻⁶ at 64 Gb/s |
| 400G Ethernet | 102 million | wrong bits, 2.4 × 10⁻⁴ at 425 Gb/s |
Hamming’s code finds the one wrong bit
1947In 1947 Richard Hamming ran jobs on Bell Labs’ Model V relay computer over nights and weekends. When it found an error it dropped the job, because it could tell something was wrong but not where. He set out to make it find the bit. His paper came out in 1950.
His smallest code adds 3 check bits to 4 data bits. Each check bit makes the number of ones in its group even. Flip any one of the 7 bits and the checks that fail spell out where it is: here checks 2 and 4 fail, and 2 + 4 = 6.
All 7 one-bit errors were tried, and each pointed at its own bit.
The same idea still guards short, fixed fields. A CSI-2 camera packet’s header carries 6 check bits for its 24, so it fixes any one wrong bit and flags any two. DDR5 memory chips fix one wrong bit in every 128 inside the chip, with 8 check bits.
Two wrong bits are past them. In a stand-in code the size of DDR5’s, 4,878 of the 9,180 two-bit errors were “fixed” into three wrong bits.
Fast links don’t get one bit wrong at a time.
Fig. 44d
| Code | Data bits | Check bits | Every one-bit error | Every two-bit error |
|---|---|---|---|---|
| Hamming (7,4), 1950 | 4 | 3 | 7 of 7 fixed | Not tried |
| Camera packet header, CSI-2 over D-PHY | 24 | 6 | 30 of 30 fixed | 435 of 435 flagged, none fixed wrong |
| DDR5, inside each memory chip | 128 | 8 | 136 of 136 fixed | 4,878 of 9,180 made into three wrong bits |
Wrong bits come in bursts
48% at a tap of 0.75At these speeds each bit leaves a tail on the next sample. A decision-feedback equalizer in the receiver subtracts it: the bit it just decided, times a tap.
When it decides a bit wrong, it subtracts the wrong amount from the next sample as well. Half the time that pushes the next sample the right way. The other half it pushes toward the wrong side, and with a tap of 0.75 that is usually enough to cross.
In a model where noise alone gets one bit in 10,000 wrong, one wrong bit made the next one wrong 48% of the time. At a tap of 0.5 the push lands right on the line, and the next bit was wrong 25% of the time.
Over 800 million bits it made 79,688 bursts. Most were a bit or two long. The longest ran 16 bits.
The chain gets longer as the tap grows. At a tap of 0.2 a wrong bit made the next wrong 0.7% of the time.
So a code for a fast link has to fix bursts. Reed-Solomon codes do it by counting symbols instead of bits.
The numbers
Tap 0.75: 79,688 bursts in 800 million bits.
| Wrong bits in a row | Bursts |
|---|---|
| 1 | 41,216 |
| 2 | 19,822 |
| 3 | 9,606 |
| 4 | 4,678 |
| 5 | 2,272 |
| 6 | 1,064 |
| 7 | 535 |
| 8 | 265 |
| 9 | 129 |
| 10 | 40 |
| 11 | 34 |
| 12 | 12 |
| 13 | 10 |
| 14 | 2 |
| 15 | 2 |
| 16 | 1 |
The numbers
Every tap, the same noise and the same 800 million bits each.
| Tap | After one wrong bit, the next is wrong | Longest burst |
|---|---|---|
| 0.2 | 0.7% | 3 |
| 0.3 | 3.5% | 4 |
| 0.4 | 11.5% | 7 |
| 0.5 | 25.0% | 9 |
| 0.6 | 38.7% | 12 |
| 0.75 | 48.3% | 16 |
| 0.9 | 49.8% | 15 |
Reed-Solomon counts symbols, not bits
1960In 1960 Irving Reed and Gustave Solomon, at MIT Lincoln Laboratory, published codes that treat a block of data as points on a curve. The sender adds more points from the same curve.
If a few points arrive damaged, the good ones still pin down the curve, and the damaged ones go back where it says. Give a code 2 extra symbols for each bad one it should fix, and it fixes that many, wherever they sit.
Every code on this page was built and given that many bad symbols at random, 300 times each. It fixed every one.
A symbol is a group of 8 or 10 bits. It costs one fix whether one of its bits is wrong or all of them.
So a burst costs only the symbols it touches. A code that fixes t symbols of m bits is sure to fix any burst of (t − 1)m + 1 bits, wherever it starts.
Every code here was given bursts of every length at every start position, and each one’s longest sure fix matched that sum. USB4’s code fixes 2 bytes, so 9 bits for sure. Its equalizer’s bursts ran past that.
USB4 turns a burst into two wrong bits
4,024 of 4,024 burstsUSB4’s code at 20 Gb/s a lane is RS(198,194): 194 data bytes, 4 check bytes, 2 fixed. With the equalizer above, one burst in about 3,000 lost a block.
So USB4 sends each bit as the data bit XOR the bit sent before it. The far end XORs each bit it receives with the one before. Inside a run of wrong bits both neighbors are wrong and the flips cancel. Only the first wrong bit and the bit just after the run come out wrong.
With the precoder on, every one of 4,024 bursts in 40 million bits left exactly 2 wrong data bits, and no block was lost. The cost: a lone wrong bit also becomes two.
DisplayPort added correction with compression
2016DisplayPort 1.4, in 2016, added RS(254,250) with 10-bit symbols, two blocks interleaved, sure to fix 31 bits in a row. It arrived with Display Stream Compression and is required whenever compression is on. HDMI 2.1 did the same with RS(255,251), sure to fix 9.
PCIe 6 splits each flit three ways
under 2 nsPCIe 6 sends data in 256-byte flits and keeps its correction under 2 ns. The code is small: the bytes take turns in three groups, and each group can fix one bad byte with its 2 check bytes.
A burst of 16 bits touches at most 3 bytes in a row, one in each group, so each group fixes its one. PCI-SIG gives 16 bits. In the model every burst up to 17 bits was fixed.
Two bad bytes in one group are one too many. Given two, the group’s code reported a fix and handed back wrong data 33% of the time: 13,081 of 40,000 tries. The rest gave up.
So each flit also carries an 8-byte CRC, checked after the correction, and a flit that fails is asked for again. PCI-SIG puts that at 5 × 10⁻⁶ of flits. Note 28 follows the CRC.
The big codes here never fixed wrong: 400G Ethernet’s gave up all 4,000 times it was given 16 bad symbols. Ethernet can afford a big code, at about 100 to 200 ns of delay.
The numbers
Bad symbols at random places in random blocks, one more than each code fixes.
| Code | Given | Tries | Gave up | Wrong fix | Share |
|---|---|---|---|---|---|
| HDMI 2.1 | 3 bad bytes | 40,000 | 20,413 | 19,587 | 49.0% |
| PCIe 6, one group | 2 bad bytes | 40,000 | 26,919 | 13,081 | 32.7% |
| USB4 | 3 bad bytes | 40,000 | 28,428 | 11,572 | 28.9% |
| DisplayPort 1.4 | 3 bad symbols | 40,000 | 38,779 | 1,221 | 3.1% |
| CD, first code | 3 bad bytes | 40,000 | 39,769 | 231 | 0.6% |
| 100G Ethernet, two-level | 8 bad symbols | 8,000 | 8,000 | 0 | 0.0% |
| Thunderbolt 5 | 13 bad symbols | 4,000 | 4,000 | 0 | 0.0% |
| 400G Ethernet | 16 bad symbols | 4,000 | 4,000 | 0 | 0.0% |
400G Ethernet’s margin, and its cliff
1.30 of 15400G Ethernet’s code is RS(544,514): 514 data symbols and 30 check symbols of 10 bits, fixing any 15.
At the limit a block carries 1.30 bad symbols on average, and 27% arrive with none. A block with 16 or more is lost: one in 1.2 trillion. At 78.1 million blocks a second, that is one every 4.36 hours.
The spec’s limit comes with a condition: the errors must be random enough that 64-byte frames are lost less than 1.7 times in 10¹². The numbers here assume random errors.
The numbers
Random wrong bits at 2.4 × 10⁻⁴; a 10-bit symbol is bad if any of its bits is. Mean 1.30.
| Bad symbols | Share of blocks |
|---|---|
| 0 | 27.1% |
| 1 | 35.4% |
| 2 | 23.1% |
| 3 | 10.0% |
| 4 | 3.3% |
| 5 | 0.8% |
| 6 | 0.2% |
| 7 | 0.000338 |
| 8 | 5.45e-05 |
| 9 | 7.79e-06 |
| 10 | 1e-06 |
| 11 | 1.17e-07 |
| 12 | 1.25e-08 |
| 13 | 1.23e-09 |
| 14 | 1.12e-10 |
| 15 | 9.5e-12 |
| 16 | 7.55e-13 |
| 17 | 5.63e-14 |
| 18 | 3.96e-15 |
| 19 | 2.64e-16 |
| 20 | 1.66e-17 |
The margin is steep on both sides. Half the errors lose 35,900 times fewer blocks. Twice the errors lose 19,700 times more, and four times the errors 119 million times more.
The real decoder, run on random blocks with random wrong bits, landed on the count at every error rate where it lost enough blocks to count.
100G two-level Ethernet’s code fixes 7 symbols in 528. At 400G’s limit it would lose 5.2 × 10−5 of its blocks.
The numbers
400G Ethernet's code, counted for random wrong bits.
| Errors | Raw rate | Blocks lost | Against the limit |
|---|---|---|---|
| ½× | 1.2 × 10−4 | 2.3 × 10−17 | 1/35,857 |
| 1×, the limit | 2.4 × 10−4 | 8.2 × 10−13 | 1× |
| 2× | 4.8 × 10−4 | 1.6 × 10−8 | 19,701× |
| 3× | 7.2 × 10−4 | 3.2 × 10−6 | 3,914,503× |
| 4× | 9.6 × 10−4 | 9.7 × 10−5 | 119,001,242× |
| 5× | 1.2 × 10−3 | 1.1 × 10−3 | 1,298,863,738× |
| 8× | 1.9 × 10−3 | 6.0 × 10−2 | 74,043,369,243× |
The numbers
The real decoder on random blocks with random wrong bits.
| Run | Blocks | Lost | Share lost | The count says |
|---|---|---|---|---|
| 400G at 2.4 × 10−4 | 4,000,000 | 0 | 0 | 8.15 × 10−13 |
| 400G at 8.0 × 10−4 | 40,000,000 | 460 | 1.15 × 10−5 | 1.16 × 10−5 |
| 400G at 1.0 × 10−3 | 8,000,000 | 1,222 | 1.53 × 10−4 | 1.53 × 10−4 |
| 400G at 1.4 × 10−3 | 1,000,000 | 4,661 | 4.66 × 10−3 | 4.70 × 10−3 |
| 400G at 2.0 × 10−3 | 200,000 | 15,533 | 7.77 × 10−2 | 7.94 × 10−2 |
| 100G at 1.0 × 10−4 | 40,000,000 | 4 | 1.00 × 10−7 | 8.93 × 10−8 |
| 100G at 2.4 × 10−4 | 4,000,000 | 213 | 5.32 × 10−5 | 5.15 × 10−5 |
| 100G at 5.0 × 10−4 | 400,000 | 2,245 | 5.61 × 10−3 | 5.61 × 10−3 |
A CD spreads each word over 109 frames
3,856 bitsThe biggest bursts belong to a disc. The CD, from 1982, uses two Reed-Solomon codes with a shuffle between them.
In the model, the first code, RS(32,28), fixes a frame with one bad byte and flags any worse. The 28 bytes it passes on are spread along the track, each byte of a word 4 frames after the one before. The second, RS(28,24), fills in up to 4 flagged bytes.
A scratch 16 frames long hits only 4 of a word’s 28 bytes, and the second code fills them in. In the model every burst up to 3,856 bits was fixed. The published figure is about 4,000 bits, 2.5 mm of track.
How big an error each link can fix
1 to 3,856 bitsSide by side, the longest burst each code is sure to fix, wherever it lands. At its luckiest start each reaches a little further: USB4 16 bits, 400G Ethernet 150, the CD 4,112. Note 26 covers the line codes these blocks ride on.
The numbers
Bits in a row, first and last wrong. Sure: fixed at every start position tried. At best: the longest fixed at its luckiest start.
| Link | Code | Symbol bits | Fixes | Sure to fix | At best |
|---|---|---|---|---|---|
| Camera packet header | 6 check bits on 24 | 1 | 1 bit | 1 | 1 |
| USB4, 20 and 40 Gb/s | RS(198,194) | 8 | 2 bytes | 9 | 16 |
| PCIe 6 | 3 groups, 2 check bytes each | 8 | 1 byte a group | 17 | 24 |
| DisplayPort 1.4 | RS(254,250), two interleaved | 10 | 2 symbols each | 31 | 40 |
| 100G Ethernet, two-level | RS(528,514) | 10 | 7 symbols | 61 | 70 |
| Thunderbolt 5 (USB4 v2) | RS(504,480) | 11 | 12 symbols | 122 | 132 |
| 400G Ethernet | RS(544,514) | 10 | 15 symbols | 141 | 150 |
| CD | RS(32,28) and RS(28,24), spread | 8 | 4 frames of the 28 | 3,856 | 4,112 |
Conditions
- Simulated
- Every count of fixed, lost and wrongly fixed blocks on this page comes from codes and decoders written for it. None was measured on hardware.
- Codes
- Reed-Solomon over 8-, 10- and 11-bit symbols, decoded by Berlekamp-Massey, Chien search and Forney. USB4’s RS(198,194) and RS(504,480) reproduce the spec’s generator polynomials and its worked example blocks. The others are built from their published sizes.
- Bursts
- A burst of L bits has its first and last bits wrong. At every start position, four patterns: every bit wrong, only the two ends wrong, and two random fills between them. “Sure” means fixed at every start; “at best” is the longest fixed at any start. Every result matched (t − 1)m + 1 and tm, with interleaving multiplying t.
- Stand-ins
- PCIe 6’s group code is a Reed-Solomon code with 2 check bytes of each group’s size, byte i in group i mod 3. It is not the spec’s matrix, and the spec’s group sizes weren’t read. The camera header’s check-bit table was written from memory and tested for its structure only: every one-bit error fixed, every two-bit error flagged. DDR5’s code is a random single-error-correcting (136,128) code. The maker’s matrix sets how two-bit errors split.
- CD
- Both codes and the 4-frame delays between them. The first code fixes one bad byte and flags the frame otherwise; the second fills up to 4 flagged bytes. No EFM, no one-frame delays and no interpolation.
- 400G at its limit
- Random, independent wrong bits at 2.4 × 10⁻⁴. A 10-bit symbol is bad if any of its bits is. Counted exactly, and checked with the real decoder in Monte Carlo runs at the rates in Fig. 44m.
- Equalizer
- Each sample is the bit (±1) plus 0.75 of the bit before plus Gaussian noise of 0.27 rms, enough alone for about one wrong bit in 10⁴. One feedback tap subtracts 0.75 of the last decision. 800 million bits at each tap from 0.2 to 0.9.
- Precoder
- USB4’s: each bit sent is the data bit XOR the last bit sent, undone at the far end by XORing each received bit with the one before. 40 million bits through the same equalizer at a tap of 0.75.
- Published
- Every error limit, line rate, delay and date comes from the specs and papers in Sources.
Models of each code. Nothing on this page was measured on a bench.
Limits
- Random errors only, except for the equalizer’s bursts. Real links mix the two, and their bursts depend on the receiver.
- A real 400G link interleaves two blocks across its lanes, so a burst on one lane spreads further than in one block. The page quotes one block.
- The equalizer is a one-tap model with made-up noise, not any real receiver.
- PCIe 6’s groups, the camera header’s table and DDR5’s matrix are stand-ins of the right size.
- The CD model leaves out EFM, the one-frame delays and the player’s interpolation, so it lands under the published 4,000 bits.
- What one wrong bit does to a compressed DisplayPort picture wasn’t run.
- Soft-decision codes such as 10GBASE-T’s LDPC aren’t covered.
Files
- sim/rs.pyGalois fields and Reed-Solomon codes: encoder, Berlekamp-Massey decoder, and every link’s code
- sim/codecs.pyEach link’s framing, bursts at every length and start, Hamming (7,4), the camera header, the DDR5-size code, the CD model, and the checks against USB4’s example blocks
- sim/waterfall.py400G at its limit, lost blocks against the error rate, the Monte Carlo runs, frames with no correction, and one bad symbol too many
- sim/dfe.pyThe one-tap equalizer’s bursts at each tap, and the precoder run
Sources
- Keysight, Overcome Forward Error Correction Challenges: IEEE 802.3 Clause 122’s bit error ratio of 2.4 × 10⁻⁴, and its condition on random errors
- Wang, IEEE 802.3 Beyond 400G, March 2021: 2.4 × 10⁻⁴ on the optical link
- Signal Integrity Journal, Gb/s Ethernet Forward Error Correction Analysis: RS(528,514) and RS(544,514), and about 100 to 200 ns of delay
- Zhang, IEEE 802.3ck, September 2018: equalizer error propagation and precoding
- Das Sharma, PCI-SIG blog: 10⁻¹² for PCIe 1.0 to 5.0, PCIe 6.0’s first-bit error rate of 10⁻⁶, correction under 2 ns, any burst up to 16 bits, the flit’s CRC and its retry rate of 5 × 10⁻⁶ (archived copy)
- Das Sharma, PCIe 6.0 Specification and Beyond, SNIA SDC 2022: the 256-byte flit and its three-way interleaved correction
- USB-IF, USB4 Specification, Version 2.0: section 3.1.1.2 (the precoder), section 4.3.1.6 and Tables A-5 and A-6 (RS(198,194) and its example blocks), sections 4.3.2.3 and 4.3.2.8 (RS(504,480)), section 3.2 (10⁻⁸ before correction at 25.6 GBd)
- Semiconductor Engineering, Video Compression And Forward Error Correction On Display Interfaces: correction required with DSC on DisplayPort 1.4 and HDMI 2.1
- Display Daily, DisplayPort 1.4 supports error correction (2016)
- Rambus, DisplayPort 1.4 FEC receiver brief: RS(254,250) with 10-bit symbols, interleaved
- Rambus, HDMI 2.1 FEC receiver brief: RS(255,251)
- TI, DS90UB953-Q1 datasheet: the CSI-2 packet header’s check bits, one bit fixed and two detected
- Patel et al., Understanding and Modeling On-Die Error Correction in Modern DRAM, DSN 2019: a two-bit error “fixed” into three
- Synopsys, Error Correction Code in DDR Memories: DDR5’s on-die correction, 8 check bits per 128
- IEEE Computer Society, Richard Wesley Hamming: the Model V and his 1950 paper
- BRG, The Great Fixer of Information Coding Errors: the dropped jobs on nights and weekends
- Reed and Solomon, Polynomial Codes over Certain Finite Fields, Journal of the Society for Industrial and Applied Mathematics, 1960
- Wikipedia, Cross-interleaved Reed-Solomon coding: the CD’s two codes and its burst of about 4,000 bits
- Stereophile, Analog Compact Disc, page 5: about 2.5 mm of track
Like this note?
Rev 1 ·
- Rev 1First published
Each fix gets a line here, with the reader's name if they want credit.