The Inference Shift: Why AI’s Center of Gravity Is Moving From Training to Inference

Inference spending passed training spending in 2025, and the vendor forecasts published since have widened that gap rather than closed it. Enterprises have stopped buying compute mainly to build models, and they now buy compute to run models continuously for millions of users.

Inference is the ongoing work of running a trained model to answer real user requests. Training, by contrast, is the one-time process of building the model. That distinction sounds academic until it reaches a budget, because one workload has an end date and the other does not. Production serving now sets the shape of compute demand, and it has no end date to plan against.

Has inference spending really overtaken training spending?

Yes. According to a Gartner report cited by CIO Dive, global spending on inference reached $23.3 billion in 2025, overtaking the $19 billion spent on training. Gartner analyst Hardeep Singh told CIO Dive that the shift shows AI adoption becoming more mainstream and production oriented. Deloitte estimated in a November report cited by Computerworld that inference already accounted for half of all AI compute in 2025. That share was projected to reach two thirds during 2026, the year now almost behind us.

Hardware vendors read the same curve from the order book. At CES 2026 in January, Lenovo executive Ashley Gorakhpurwalla told Computerworld that his company now forecasts an 80/20 split favoring inference over training. That ratio reverses the mix that defined the last buildout cycle. It also comes from a vendor counting shipped units rather than modeling a market from the outside.

Two independent spending estimates and one vendor forecast land on the same inflection point. The crossover therefore deserves treatment as an observed condition rather than a projection.

For three years, infrastructure planning ran on a training-first assumption: large upfront capital, then a quiet stretch before the next model generation. That pattern no longer describes how compute gets consumed. Serving live requests accounts for most GPU hours in a typical enterprise fleet today.

Training is an event. Inference is a utility.

The two workloads behave differently at every level of the stack. A training run has a start date and an end date. Teams assemble a cluster, drive a model to convergence over weeks, then tear it down or repurpose it. Inference behaves more like a power draw. It runs every hour the product stays live, and it grows with every new seat and every new workflow.

Research from Sevilla et al., in “Compute Trends Across Three Eras of Machine Learning” (arXiv, 2022), documents that training compute for the largest models doubled roughly every six months after the deep learning era began. Before 2010, training compute grew in line with Moore’s Law. That curve describes a small number of frontier labs racing each other. It says little about the thousands of companies deploying finished models into support desks and internal agents.

How the two workloads differ on paper

The table below sets them side by side on the dimensions that drive procurement decisions.

Read the last row first, because it governs the rest. A training-driven purchase follows an internal schedule that engineering controls. An inference-driven purchase follows customer behavior, which no planning committee controls.

Enterprises that size infrastructure with a training mindset overbuild for peak and underbuild for sustained load. Singh flagged exactly this to CIO Dive, describing demand moving from foundation model training toward continuous inference embedded in applications and business processes. As a result, a vendor shortlist built on training benchmarks measures the wrong workload. Axe Compute’s guidance on how to evaluate a GPU cloud provider makes the same argument in operational terms.

Agentic requests raise the cost of a single query

More users sending more requests is only half the story. Each request can also burn more compute than it did a year ago. Snell et al., in “Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters” (arXiv, 2024), show that letting a model spend additional compute at inference time improves performance on hard prompts. In some cases, that approach beats simply training a larger model.

The finding reframes the buying question. Some of the highest value workloads deliberately spend more compute per query through reasoning chains and verification passes. Consequently, inference capacity must absorb variable and sometimes heavy per-request load rather than a flat curve. Agentic systems sharpen the problem, since one user click can trigger dozens of chained model calls. Sustained serving capacity then sets the limit on what a product can ship.

Serving is a memory bandwidth problem before it is a FLOPS problem

Here is where training intuition fails hardest. Token generation reads the model weights out of high bandwidth memory for every token produced, which makes the decode phase bound by memory bandwidth rather than arithmetic throughput. Prefill, the pass that processes the incoming prompt, behaves the opposite way and saturates compute. A single request stresses two different parts of the same accelerator in sequence.

Context length compounds this. The key-value cache grows with every token in the conversation and with every concurrent user, and it competes with model weights for the same memory. Long agentic sessions consume far more memory per user than a short chat turn. In practice, memory capacity often caps concurrency long before compute does.

Why batching does not solve it for you

Continuous batching raises throughput by packing more requests through the same weight read. However, it also lengthens the queue that any single request waits in, which pushes out tail latency. Serving engineers spend their time on that tradeoff, tuning batch size and scheduling policy against a latency target the product team set.

None of this shows up on a datasheet comparison of peak floating point performance. Accelerator selection for serving turns on memory bandwidth, memory capacity per device, and how the interconnect handles a sharded model under load. Axe Compute’s approach to hardware choice starts from the workload profile for that reason.

Cheaper tokens, larger bills

Unit economics improved fast. The Stanford HAI 2025 AI Index Report found that the inference cost of a system performing at GPT-3.5 level fell more than 280-fold between November 2022 and October 2024. That report attributes part of the decline to hardware costs falling roughly 30% a year, and it credits energy efficiency gains of around 40% a year.

Falling cost per token does not shrink total inference spending. In practice, it raises it. Cheaper inference makes new categories of work economically viable. A support workflow that once cost too much per ticket to automate becomes affordable. Elsewhere, a coding agent that ran too slowly to be useful becomes fast enough to ship. Each new use case adds recurring load on top of what already existed.

Volume therefore grows faster than unit cost falls. Gartner’s forecast reflects that arithmetic, with inference spending climbing even as hardware and software efficiency keep improving. Computing has repeated this pattern for decades, because software expands to fill whatever silicon gets cheaper.

The budgeting error this creates

Treating inference as a fixed line item tied to today’s usage is the most common planning mistake. Teams that do it run short inside a year. A better model treats serving capacity as a growth line that tracks product adoption.

Contract structure follows from that. Enterprises locked into rigid, single-region, training-style agreements struggle to add capacity when a product takes off. By contrast, capacity that expands within days and across regions matches how inference demand actually arrives.

What the shift changes about how you buy compute

Once inference dominates the mix, several line items in an infrastructure budget behave differently.

Procurement horizons shrink

Training capacity can be reserved for a defined window, because the workload has a known start and end. Inference capacity has neither. So the buying horizon compresses from quarters to days. Multi-month lead times, standard in traditional GPU procurement, do not fit a workload that must grow with next month’s traffic. Axe Compute’s analysis of why enterprise GPU procurement is broken documents how far that gap has widened. A year-long wait for hardware becomes a product decision the moment usage outruns the forecast.

Geography becomes a latency decision

Training clusters can sit wherever power and land are cheapest. Output ships once, at the end of the run, so a few hundred milliseconds of network distance costs nothing. Live inference works under different physics. A model answering a support ticket in Tokyo should not wait on a round trip to another hemisphere. As a result, buyers need capacity spread across regions rather than concentrated in one campus. Agentic workflows amplify the penalty, since latency multiplies across every chained call.

Utilization replaces peak FLOPS as the benchmark

Peak throughput measures how fast a cluster can finish a training run. Sustained utilization measures whether a fleet can answer requests at three in the morning during a traffic spike. Those are different engineering problems. A cluster that sets a record for six weeks and then idles is a poor template for continuous serving. Consequently, provisioning speed and measured utilization under variable load now matter as much as chip specifications on a datasheet.

Buildout planning is still running on a training cadence

Data center construction happens in large discrete phases. Inference demand does not. It grows with every product launch and with every workflow that quietly starts calling a model. Meanwhile, the capacity meant to serve that demand gets financed and built around a training-era mental model, where one large facility comes online to serve one large run.

Financing cycles measure in years. Product adoption curves measure in weeks. A serving fleet sized against a construction schedule will therefore sit idle in one quarter and run hot in the next, and neither state is recoverable after the fact. That gap creates an opening for infrastructure that scales with usage instead.

Deloitte’s projection that inference reaches two thirds of AI compute in 2026 has been running against real order books all year. Teams now drafting 2027 capacity plans on training assumptions will meet the mismatch in production, under load, with customers watching.

The questions worth asking before your 2027 planning cycle

Training infrastructure does not disappear. Frontier labs will keep building larger clusters, and the doubling trend documented by Sevilla et al. still has room to run. For most enterprises, though, the urgent question has changed from how to train a model to how to serve one everywhere users are.

Start with the split. How much of current spend is tied to a fixed training schedule, and how much tracks ongoing serving load? Most teams have never separated the two on a single sheet. The exercise usually surprises them.

Then check distance. Does existing capacity sit close enough to users to survive a multi-step agentic interaction without visible lag? Measure against the slowest chained call in the trace, since that call sets what the user perceives.

Finally, test elasticity. Can additional capacity come online in days if usage outruns the forecast? A team that cannot answer yes is running a product roadmap on a procurement calendar.

The next cycle belongs to operators who keep inference running close to their users and can add capacity in the same week that traffic arrives. Capacity planning is becoming a product discipline, and the teams treating it that way will ship features their competitors have to postpone.

About Axe Compute

Axe Compute Inc. (NASDAQ: AGPU) is a neocloud AI infrastructure platform built on a fundamental premise: AI innovation should not be constrained by hardware choice or inventory limitations. Axe Compute gives enterprises and AI innovators choice across hardware, geography, and deployment speed through two delivery models: Axe Compute Access, providing the latest GPU compute options in as fast as 48 hours across numerous global locations, and Axe Compute Build, enabling enterprises to access large-scale dedicated AI factories, all backed by enterprise-grade SLAs and support.

Frequently Asked Questions

Why is inference spending overtaking training spending?

Training is a one time capital event tied to building a model, while inference runs continuously as long as a product is live. As more companies move from pilot projects to shipped AI products, the ongoing cost of running models accumulates faster than the periodic cost of training them.

Does falling inference cost per token mean total inference spending will fall too?

No. Falling cost per token tends to increase total spending because it makes more applications economically viable, which increases the volume of inference requests faster than unit costs fall.

What is test-time compute and why does it matter for infrastructure buying?

Test-time compute refers to using additional compute during inference, such as extended reasoning steps, to improve a model’s output on a given prompt. It matters for infrastructure because some inference workloads consume significantly more compute per request than simple lookups, increasing variability in serving loads.

Should enterprises still invest in dedicated training clusters?

Large model builders still need dedicated training capacity for defined periods. However, most enterprises deploying existing models need infrastructure sized for continuous inference rather than periodic training runs.

How does geography affect inference infrastructure decisions?

Inference often serves live user traffic, so latency depends on how close compute sits to end users. Training output ships once a model finishes, so training capacity is less sensitive to physical location.

What is the main risk of sizing infrastructure with a training mindset?

Teams that plan for large, short bursts of peak throughput tend to over-provision for training-style peaks and under-provision for the steady, variable, round-the-clock demand that inference actually generates.

About Axe Compute

Sources

• Makenzie Holland, “AI infrastructure spending shifts in latest sign of deployment maturity”, CIO Dive
• Computerworld staff, “CES 2026: AI compute sees a shift from training to inference”, Computerworld
• Stanford Institute for Human-Centered Artificial Intelligence (HAI), “The 2025 AI Index Report”, Stanford HAI
• Jaime Sevilla, Lennart Heim, Anson Ho, Tamay Besiroglu, Marius Hobbhahn, Pablo Villalobos, “Compute Trends Across Three Eras of Machine Learning”, arXiv
• Charlie Snell, Jaehoon Lee, Kelvin Xu, Aviral Kumar, “Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters”, arXiv