Peak FLOPS tells you almost nothing about how a GPU will perform on your workload. Two frontier models running on comparable hardware can differ by more than double in delivered efficiency.
Model FLOPs Utilization (MFU), the ratio of achieved throughput to theoretical peak, is the metric that closes this gap. Google’s PaLM paper reported an MFU of 46.2% for its 540 billion parameter model, more than double the 21.3% it reported for GPT-3. Buyers who benchmark on peak FLOPS alone routinely pay for capacity their workload can never actually reach.
Model FLOPs Utilization (MFU) is the percentage of a GPU’s theoretical peak compute that a model actually uses during a real training or inference run. It is the most reliable number for comparing GPUs across vendors, because it measures delivered performance on your workload instead of a marketing ceiling.
Buyers who shop on spec sheets pay for capacity they never use. The fix is simple, and slightly uncomfortable. Benchmark your own model, then hold the provider to the result in writing.
Peak FLOPS Describes a Limit You Will Not Reach
Every spec sheet leads with a figure measured under ideal conditions: perfect batch size, no memory stalls, no kernel left waiting. Google’s TPU v4 research, published through the Association for Computing Machinery, takes on that habit directly. Norm Jouppi and his co-authors wrote that “many in the ML community think peak FLOPS/second are a good performance proxy.” Their measured results argued against that assumption. In practice, the distance between the rated figure and the delivered figure is where procurement goes wrong.
The PaLM paper, “PaLM: Scaling Language Modeling with Pathways” by Aakanksha Chowdhery and co-authors, puts a number on that distance. It reports a Model FLOPs Utilization of 46.2 percent for its 540 billion parameter model, measured with self-attention included. That same paper places GPT-3 at 21.3 percent, based on figures OpenAI reported. Two frontier labs of a similar era therefore differ by more than double on one measure. Neither figure reflects weak engineering.
The same paper also shows why the metric matters. PaLM ran at 57.8 percent hardware FLOPs utilization on TPU v4, yet only 46.2 percent MFU. The difference is extra recomputation the hardware performed to save memory, which kept the chips busy without moving the model forward. For that reason, the authors proposed MFU as the more honest, implementation-independent measure.
Where the Missing Capacity Goes
Memory bandwidth caps how fast weights and activations reach the compute units. Meanwhile, collective communication stalls the pipeline while gradients synchronize across devices. Smaller effects stack on top of those two: kernel launch gaps, padding waste, pipeline bubbles at the start and end of every step. None of this appears on a spec sheet. All of it appears in your training logs.
Treat the Headline Figure as a Ceiling
Teams building at that scale land between roughly 20 and 46 percent, according to the PaLM paper’s own comparison. Any buyer evaluating a rack for a narrower workload should therefore expect at least as much variance. A GPU rated at 1,000 peak teraflops does not deliver 1,000 teraflops to your model. Instead, it delivers whatever fraction your architecture and parallelism strategy can extract at your batch size. That fraction sets your real cost per token.
What Is Model FLOPs Utilization and Why Does It Matter?
MFU compares what a model’s architecture requires against what the hardware actually delivers during a run. In effect, it strips out marketing language and reduces the comparison to one number.
Dashboard utilization answers a different question. A monitoring tool reports whether a chip is busy. By contrast, MFU asks whether the chip is busy doing useful work for your model. A GPU can show 95 percent utilization on a dashboard while running at 20 percent MFU, because it stays fully occupied moving data rather than computing.
Three questions turn MFU from a research curiosity into a purchasing decision.
What MFU Does the Provider Report on a Model Close to Yours?
MFU varies by model family as well as by chip. For example, a dense transformer and a mixture-of-experts model land in very different places on identical hardware, because MoE routing introduces communication overhead that dense models never carry. Therefore, ask any provider for MFU on the closest published analog to your workload. A generic figure pulled from an unrelated benchmark is not evidence. Our guide on evaluating GPU cloud providers walks through the questions worth raising in that conversation.
What Does Utilization Look Like at Your Actual Batch Size?
MFU improves with larger batch sizes up to a point. After that, communication overhead across GPUs begins to dominate and the curve flattens. Consequently, a benchmark run at the vendor’s preferred batch size may not reflect your production traffic. Request the efficiency curve across a range of batch sizes instead of the single best point on it.
How Much Theoretical Gain Survives Your Interconnect?
A newer GPU generation with more FLOPS on paper only helps if the interconnect and memory bandwidth can feed it fast enough. Otherwise the extra capacity sits idle. For that reason, generational upgrades sometimes deliver less real improvement than the marketing headline suggests. Our NVIDIA Blackwell GPU comparison breaks the gain down by workload type rather than quoting a single multiplier, because the multiplier moves with what you actually run.
What MLPerf Tells You, and What It Does Not
MLCommons runs the MLPerf benchmark suite, the closest thing the industry has to a standardized comparison across GPU generations and vendors. Every submission runs the same defined tasks under audited rules. As a result, MLPerf numbers carry far more weight than vendor-reported peak FLOPS. However, the suite measures a fixed set of reference models and datasets. Your production workload is rarely identical to any of them.
NVIDIA’s own reporting shows both the value and the limit. NVIDIA’s technical blog on MLPerf Training v5.0 states that its GB200 NVL72 system delivered performance “up to 2.6x more” per GPU compared to the previous Hopper generation. That figure is real, audited, and reproducible. Even so, the phrase “up to” is doing heavy lifting, since the multiplier applies to the specific model and configuration used in the submission.
Audited Results Still Hide a Wide Spread
MLCommons results on the inference side reinforce the pattern. The MLCommons release covering MLPerf Inference v5.0, published in April 2025, reported 17,457 individual performance results from 23 submitting organizations. Those submissions spanned dozens of hardware and software combinations tested against the same reference models. Such a spread, measured against identical tasks, shows how much the surrounding software stack changes the real number.
Precision adds another variable. Newer hardware supports 4-bit formats such as NVFP4, which NVIDIA says can raise throughput with accuracy loss of about one percent or less on the models it tested. Meanwhile, the accuracy tradeoffs and software maturity behind that gain vary by workload. Ask to see the number applied to something close to your own model first.
Reading a Benchmark Table Without Getting Misled
The table below shows how three common evaluation methods differ in what they measure.
| Method | What It Measures | What It Misses |
|---|---|---|
| Peak FLOPS (spec sheet) | Theoretical maximum under ideal conditions | Real batch size, memory bandwidth, communication overhead |
| MLPerf benchmark result | Standardized task performance, audited and reproducible | Whether your model architecture resembles the reference task |
| Model FLOPs Utilization (MFU) | Actual compute delivered relative to your model’s requirements | Cost and availability at the scale you plan to run |
No single row gives a complete picture, so a rigorous evaluation layers all three. Peak FLOPS sets the theoretical ceiling. MLPerf provides an audited comparison against competing hardware. Finally, MFU on a workload close to yours estimates what production will actually return.
Where Buyers Get This Wrong in Practice
Most procurement mistakes trace back to comparing numbers that were never measured the same way. A vendor quoting peak FLOPS against a competitor’s MLPerf result is not making a fair comparison, even when both figures sit side by side in the same deck.
Internal engineering teams make a related error. They benchmark on a small proof-of-concept model and extrapolate to production scale. Later, they discover the efficiency curve does not hold. Small models frequently show artificially high MFU, because they carry less communication overhead relative to their compute. That pattern reverses once the same architecture scales up.
The Software Stack Moves the Number as Much as the Chip
Driver version, collective communication library, compiler settings, and parallelism configuration all shift delivered throughput on identical silicon. Indeed, the MLPerf inference spread of 17,457 results across 23 organizations is partly a record of exactly that. Two teams with the same accelerator can land far apart because one tuned the stack and the other accepted defaults. Ask which software versions produced any number a vendor shows you. Then ask whether your team gets the same stack on day one.
Generational Claims Deserve the Same Scrutiny
A new chip generation typically improves peak FLOPS substantially between releases. However, the real gain for a specific workload can lag well behind that headline. Memory bandwidth and interconnect do not always scale proportionally, so the bottleneck simply moves. Buyers who ask for workload-specific numbers make better decisions. They also avoid renegotiating contracts six months into a deployment.
The Questions to Run Before You Commit
Buyers who get burned on GPU procurement almost always skipped one of four steps before signing.
Confirm What Was Measured, and How
First, ask what workload the vendor’s benchmark actually ran. A number attached to image classification tells you little about a transformer training job. Similarly, a number pulled from a 7 billion parameter model tells you little about a mixture-of-experts deployment at ten times the parameter count. Our analysis of the mixture-of-experts GPU shortage covers why MoE architectures need their own efficiency benchmarking.
Second, ask how the vendor measured utilization. MFU and hardware FLOPs utilization are different measures, as the PaLM figures show. A generic GPU-busy percentage from a monitoring tool is different again. A vendor citing “95 percent utilization” from a dashboard answers a different question than a vendor citing MFU from an actual training run.
Pilot the Workload, Then Check the Recourse
Third, run a short pilot on your own workload before committing to a long-term contract. Even a few hours at production scale reveals more than any published figure. The run captures your batch size and your data pipeline. More important, it exposes the communication pattern your model produces across GPUs.
Fourth, confirm what happens when the delivered number misses the benchmark. Enterprise-grade commitments matter more here than any single performance figure. A provider willing to write utilization and availability commitments into a contract expects those numbers to hold under your workload. Demo environments built to flatter the hardware carry no such exposure.
What to Instrument During the Pilot
Record step time and tokens per second per GPU, then derive MFU from those figures rather than from a dashboard reading. Capture peak memory footprint at your target sequence length. Time a checkpoint write and a restart from that checkpoint. After that, ask the provider to fail a node during the window and watch what the job does. Recovery behavior belongs in the evaluation, because a long training run will need it.
Why Contractual Commitments Beat Marketing Numbers
Every figure in this piece came from a benchmark run under specific, disclosed conditions. That transparency is what makes them trustworthy. MLCommons audits its submissions, and Google published the exact configuration behind the PaLM MFU numbers. Buyers should hold their infrastructure provider to the same standard.
A number on a website is a claim. Written into a service level agreement, with recourse when it is missed, that number becomes a commitment. Enterprises evaluating providers should therefore ask two things: what MFU the vendor reports, and what happens contractually when a workload falls short of it. Axe Compute writes its commitments into both delivery models, Access and Build, including a 99.93% uptime SLA monitored continuously with service credits, because the number a client plans against should survive contact with a production run.
The same standard applies to the surrounding operational detail. Which support tier handles a stalled job at two in the morning. How long a replacement node takes. What the escalation path looks like when throughput degrades mid-run. Those terms decide whether a benchmark result holds over a six-month training campaign.
Building a Benchmark Plan That Survives Production
A spec sheet is a starting point. The real evaluation happens when you run your own model on your own data pipeline and compare what comes back to what was promised. Teams that build this step into procurement avoid the gap between paper capacity and delivered capacity that catches everyone else off guard.
The plan itself is short. Pick the model closest to what production will run. Execute it at the batch size your traffic actually generates. Record MFU rather than dashboard utilization. Then ask the provider to commit to that result in writing, alongside the software versions that produced it.
Hardware generations will keep arriving faster than most procurement cycles can absorb them. As a result, the next release will land mid-contract for many teams. Book the pilot window before the term starts, and specify in the agreement how a new generation gets benchmarked against your workload rather than against a reference task. Axe Compute Access provides dedicated GPU capacity, ready when the workload is, on terms that start at one month, which leaves room to measure first and sign second.
Benchmark against your own workload
Frequently Asked Questions
What is Model FLOPs Utilization (MFU) and why does it matter more than peak FLOPS?
MFU measures the ratio between the compute a model architecture actually requires and the throughput hardware delivers during a real training or inference run. Peak FLOPS describes a theoretical ceiling under ideal conditions that production workloads rarely reach, so MFU gives a far more accurate picture of real-world performance.
Is a high GPU utilization percentage the same as high MFU?
No. Dashboard utilization percentages typically show whether a GPU is busy at all, including time spent waiting on memory transfers or communication. MFU specifically measures useful compute work relative to the model’s requirements, so a GPU can show high dashboard utilization while running at low MFU.
What is MLPerf and how should buyers interpret its results?
MLPerf is a standardized benchmark suite run by MLCommons that measures hardware and software performance on a fixed set of reference tasks under audited conditions. It is more trustworthy than vendor-reported peak FLOPS, but results apply to the specific models tested and may not reflect a buyer’s own workload.
Does a newer GPU generation always deliver proportionally more real-world performance?
Not automatically. Gains depend on whether the interconnect and memory bandwidth around the GPU can supply data fast enough to use the added compute. A generation with a higher peak FLOPS number can still show a smaller real-world gain if communication overhead dominates the workload.
What should a buyer ask a GPU provider before signing a contract?
Ask what workload the vendor’s benchmark actually ran, whether the reported number is MFU or a generic utilization metric, and whether a short pilot on the buyer’s own workload is possible before committing. Buyers should also ask what contractual recourse exists if the delivered performance falls short of the benchmark.
Why does MFU vary so much between different AI models on the same hardware?
MFU depends on model architecture, batch size, and parallelism strategy as much as on the underlying chip. Dense transformer models and mixture-of-experts models place very different demands on communication and memory bandwidth, which produces different efficiency numbers even on identical hardware.
How does batch size affect real-world GPU performance?
Larger batch sizes generally improve MFU up to a point by giving the GPU more work per communication step. Beyond that point, communication overhead across GPUs can cause efficiency to plateau or decline, so a benchmark run at one batch size may not represent performance at another.
About Axe Compute
Axe Compute Inc. (NASDAQ: AGPU) is a neocloud AI infrastructure platform built on a fundamental premise: AI innovation should not be constrained by hardware choice or availability. The company provides enterprises and AI innovators with flexibility across hardware, geography, and deployment models through two core offerings: Axe Compute Access, delivering a wide range of the latest high-performance GPU infrastructure across global locations, and Axe Compute Build, enabling the design, deployment, ownership, and operation of large-scale, dedicated AI infrastructure worldwide. All solutions are supported by enterprise-grade SLAs and operational expertise. Axe Compute is headquartered in Pittsburgh, Pennsylvania. For more information, visit axecompute.com or contact us at info@axecompute.com.
Sources
- Aakanksha Chowdhery et al., “PaLM: Scaling Language Modeling with Pathways”, arXiv: 46.2% MFU and 57.8% HFU for PaLM 540B on TPU v4; GPT-3 at 21.3% MFU
- Norman P. Jouppi et al., “TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings”, ISCA 2023, ACM
- NVIDIA, “NVIDIA Blackwell Delivers up to 2.6x Higher Performance in MLPerf Training v5.0”, NVIDIA Technical Blog
- MLCommons, “MLCommons Releases New MLPerf Inference v5.0 Benchmark Results”, April 2025
- NVIDIA, “Introducing NVFP4 for Efficient and Accurate Low-Precision Inference”, NVIDIA Technical Blog: 1% or less accuracy degradation versus FP8 on DeepSeek-R1-0528