Block-Scaled MXFP4 With Stochastic Hadamard Rounding Matches BF16 Convergence on 70B Pre-Training

Machine learning numerics researchers and hardware architects across San Francisco, Santa Clara, and Stanford University released an open technical study and Triton/CUDA kernel library on Friday morning validating native 4-bit Open Compute Microscaling (MXFP4) forward and backward matrix multiplications across a complete 2-trillion-token pre-training run of a 70-billion-parameter dense transformer. Standardized by the Open Compute Project Microscaling Formats (MX) working group committee, MXFP4 assigns a shared 8-bit power-of-two exponent scale (E8M0) to every contiguous block of thirty-two 4-bit (E2M1) floating-point elements. While 4-bit quantization is widely deployed for post-training inference weight compression, naive 4-bit activation and gradient quantization during training previously collapsed due to heavy-tailed channel outliers and quantization bias in backward weight-gradient accumulation.

To eliminate quantization bias without falling back to 8-bit precision, the engineering team fused a fast block-diagonal Walsh-Hadamard transform with hardware-assisted stochastic rounding inside the tensor-core epilogue before packing activation tiles into 4-bit registers. Rotating activation vectors in 32-element Hadamard blocks spreads isolated outlier spikes evenly across all coordinates in O(d log d) register operations, ensuring that the E2M1 dynamic range is never saturated by a single activation channel.

Evaluated on a 512-GPU liquid-cooled cluster in Santa Clara interconnected via 800-gigabit RoCEv2 fabrics, the MXFP4 training recipe achieved a 1.85x wall-clock throughput speedup over an FP8 baseline and a 2.6x speedup over BF16 while matching validation perplexity within 0.008 nats and downstream zero-shot MMLU and HumanEval coding scores within statistical noise. Only the final embedding projection, RMSNorm scales, and router logits remained in high precision.

All custom fused Hadamard-MXFP4 kernels, PyTorch torch.compile custom ops, and deterministic checkpoint evaluation scripts have been published under a permissive MIT license. Researchers can inspect the convergence logs and kernel benchmarks via the PyTorch Foundation Ecosystem Portal.