Quantized GEMM Error — Isolating the two design choices behind MXFP4 vs. NVFP4
A research paper isolating what actually explains NVIDIA's claimed NVFP4-over-MXFP4 training advantage: a from-scratch quantization engine, a numerical-error bound derived from first principles, and a 1.6-million-trial factorial study pinning the advantage on scale format rather than block size.
Quantized GEMM Error is a short paper investigating why NVIDIA's NVFP4 format needs 36% fewer training tokens than MXFP4 to match pretraining loss — I'm currently looking for a venue to submit it to — a number that can't say which of NVFP4's two design changes (a smaller 16-value block vs. MXFP4's 32, and a finer E4M3 scale vs. MXFP4's power-of-two E8M0) actually matters, since both change at once. I built a full 4-bit block-scaled quantization engine from scratch in NumPy (FP4/FP8/E8M0, block scaling, a random Hadamard transform, a quantized-GEMM pipeline), validating every layer bit-exact against reference implementations (ml_dtypes, Microsoft's microxcaling) before trusting a single result. Classical rounding-error bounds don't hold for block-scaled formats under exact accumulation, so I derived a new one from first principles and validated it against theory and an independently published result. A preregistered 640-configuration, 1.6-million-trial factorial sweep — run on CPU for exact fp64 accumulation, with checkpointing that survived a mid-run thermal shutdown — isolates the two confounded factors directly. The headline finding: scale format, not block size, is what actually drives NVFP4's advantage on a backward-error proxy, validated against real GPT-2 activations and disclosed honestly where the theory and the data disagreed.
The problem
NVIDIA's own benchmark for NVFP4 bundles two independent design changes — a smaller 16-value quantization block (vs. MXFP4's 32) and a finer-grained E4M3 scale format (vs. MXFP4's power-of-two E8M0) — into a single headline number, so a 36%-fewer-tokens claim says nothing about which change actually matters. Classical numerical-error theory (Higham's rounding bounds) doesn't apply to block-scaled quantization under exact accumulation either, so there was no principled way to even predict which factor should dominate before running the experiment.
The solution
I designed a preregistered 2×2 factorial experiment — real hardware formats plus two synthetic, non-hardware control configurations — to isolate block size from scale format, crossed with tail weight (how 'spiky' activation distributions are), rounding mode, and an outlier-spreading transform. The sweep ran 640 configurations and 1.6 million trials on a laptop CPU — deliberately, since the experiment needs exact fp64 accumulation that GPUs handle poorly — with checkpoint/resume infrastructure that survived an actual mid-run thermal shutdown without losing data. The originally preregistered comparison, locating where the derived error bound breaks down, turned out uninformative on this grid; rather than force a false positive, I disclosed that honestly and pivoted to a direct magnitude-based causal analysis: scale format, not block size, dominates NVFP4's advantage across nearly the whole tested range, with an honest caveat at the heaviest tail weight. I validated the synthetic model against real GPT-2 activations (strong transfer at 2 of 3 layers, the third traced to a documented 'massive activations' phenomenon), verified every citation against primary sources by hand, and put the paper through two adversarial review passes before submission.