OCP Microscaling MXFP4 Tensor Cores Achieve 99.6 Percent BF16 Convergence Parity on 405B Pretraining via Stochastic Rounding Hardware

Floating-point arithmetic architects and deep-learning systems researchers across Santa Clara, San Jose, and UC Berkeley published 15-trillion-token pretraining convergence benchmarks on Saturday evening through the Open Compute Project (OCP) Microscaling Formats (MX) working committee, demonstrating that hardware MXFP4 Tensor Cores equipped with inline stochastic rounding match BF16 validation loss within 0.4 percent on 405-billion-parameter transformer models while doubling raw matrix-multiply throughput over FP8. While 4-bit block floating-point formats have become standard for memory-bound inference, applying 4-bit quantization to forward activations, weights, and backward activation gradients during pretraining previously suffered from severe gradient underflow and late-stage loss spikes.

Conforming to the OCP MX v1.0 specification, the validated datapath groups every 32 contiguous E2M1 (1 sign, 2 exponent, 1 mantissa bit) elements along the inner dot-product reduction dimension under a single shared 8-bit E8M0 power-of-two block scale factor. During the backward pass, a dedicated 24-bit SplitMix linear-feedback shift register (LFSR) embedded inside each tensor-core accumulator lane injects uniform pseudo-random dither bits prior to E2M1 truncation, guaranteeing an unbiased expected value for small gradient updates that would otherwise round deterministically to zero.

To prevent inter-channel variance from saturating the 32-element block scale during weight-gradient (Wgrad) transposed GEMMs, the hardware fuse-rotates tile fragments through a 32x32 block-diagonal Hadamard crossbar inside the L1 shared-memory load unit. This orthogonal basis transform redistributes activation outlier spikes across all 32 lanes in a single clock cycle at zero tensor-core throughput penalty.

Across a 256-accelerator test cluster at 3100 Coronado Drive in Santa Clara funded by a $360,000 open-silicon research program, end-to-end 405B training step times dropped by 41 percent at 61 percent Model FLOPs Utilization (MFU). Reference SystemVerilog arithmetic units and PyTorch quantization emulators are available in the Microsoft Microxcaling GitHub Repository.