Contact Us

Latency is a geography problem

Two white tin cans connected by a single taut string spanning a wide dark navy background — illustrating that AI inference latency is determined by physical distance, not compute speed.

Key finding: Real-time AI applications, such as fraud detection, medical decision support and autonomous systems, are bottlenecked not by GPU speed but by round-trip distance to a centralized data center. Standard fiber adds 1ms per 100km. New York to London is 55ms before a model runs a single step. For a 50ms latency budget, that gap cannot be closed by model optimization. It requires compute in-region.

In case the real-time AI application is not meeting its latency target. The engineering team will dig into the model. They quantize it, tune the batch size, and profile CUDA kernels. By doing this, latency improves generally by 8%.

The target was 50 milliseconds. The round-trip distance to the inference cluster adds 60 milliseconds before the model runs a single step.

This is not an optimization problem. It is a geography problem. And treating it as the wrong kind of problem is how real-time AI initiatives fail.

1ms
Round-trip delay per 100km of fiber
<50ms
Consumer-facing AI response target
100×
Inference scale vs training — NVIDIA projection
200+
Axe Compute inference locations globally

Why AI Inference Latency Is a Geography Problem

Light in optical fiber travels at approximately 200,000 kilometers per second, roughly two-thirds of its vacuum speed due to the refractive index of silica glass. Standard single-mode fiber introduces approximately one millisecond of round-trip delay per 100 kilometers of distance — a rule derived from first principles and confirmed across engineering references including mapyourtech.com’s April 2026 analysis of fiber propagation physics.

New York to London spans approximately 5,500 kilometers by fiber. That is 55 milliseconds of round-trip propagation delay, before any processing occurs, before the model receives the first token, before a response byte returns to the client.

DataRobot’s April 2026 latency deployment analysis states it directly: in many AI systems, “the biggest latency bottleneck is not the model. It is the distance between where compute runs and where data lives.” A few extra network hops across regions, cloud boundaries, or enterprise systems add hundreds of milliseconds. That penalty repeats across every retrieval step and orchestration call in an agentic pipeline.

Training Tolerates Distance. Inference Does Not.

McKinsey’s December 2025 analysis of hyperscaler strategies, reported by Data Center Dynamics, found that training workloads tolerate delays of up to 100 milliseconds between adjacent regions. That is why large training clusters site in remote, power-rich locations: latency is irrelevant to the outcome.

Inference is the architectural opposite. Iron Mountain’s April 2026 analysis of training versus inference data center design reports that consumer-facing AI generally targets sub-50-millisecond response times, requiring inference infrastructure close to population centers with diverse connectivity.

For specific application categories, the requirement is tighter still.

Financial Risk Scoring and Fraud Detection

Sift’s February 2026 fraud detection analysis confirms that payment processors require AI decisions within 100 milliseconds to avoid transaction delays. Systems handling millions of transactions per second process each one within that window. Missing it does not produce a slow transaction. It produces a blocked one. Global payment fraud losses reached $48.2 billion in 2024, this the reason this requirement is non-negotiable.

Medical Decision Support

Clinical AI systems integrating vitals, lab results, and patient histories deliver risk scores and deterioration alerts in near real time. An alert that arrives after the clinical window has closed carries no value. The latency budget is defined by the speed of the clinical event, not a product specification. Backhauling patient data to a centralized region for inference and waiting for the round-trip return is not architecturally viable for time-critical care decisions.

Autonomous Systems and Edge AI

Equinix’s January 2026 latency analysis notes that autonomous vehicles need instant responses when sensor data indicates hazards, and connected ambulance systems use edge compute to process patient vitals during transport so hospitals can prepare for incoming emergencies. Centralized inference for safety-critical control loops is not a performance tradeoff. It is an architectural disqualifier.

Quantifying AI Inference Latency by Region

Using the verified 1ms per 100km round-trip rule, the physics-minimum propagation penalties for centralized inference from major cloud regions are calculable and consistent:

Table 1 — AI inference propagation latency by region pair

Route Round-Trip Delay Verdict at 50ms Budget
US East Coast → Western Europe 55–75ms Budget exceeded before inference starts
US East Coast → Southeast Asia 160–200ms 3–4× over budget on propagation alone
Northern Virginia → London ~110ms 2× over budget; model speed irrelevant
Single EU region serving continent 10–40ms Viable for some use cases; edge of budget for others

Sources: mapyourtech.com April 2026; AWS Switzerland region guide October 2025; Wikipedia network performance.

AWS’s October 2025 Switzerland region guide confirms cross-region routing within Europe adds single to double-digit milliseconds on top of propagation. Real-world figures exceed these physics minimums due to router hops, congestion, and device processing overhead.

For a real-time AI application targeting 50 milliseconds total response time, a 60-millisecond propagation penalty means the latency budget is exhausted before inference begins. Model speed is irrelevant.

Data Gravity Compounds the Problem

Most production inference systems are not pure language model calls. They retrieve documents, query databases, call external APIs, and pass results through multiple processing steps.

If inference runs in one region and retrieval runs in another, the system pays a latency penalty before the model starts useful work. In a RAG pipeline making three retrieval calls to a distant data store, that penalty appears three times. In an agentic workflow making five tool calls, five times.

datacenters.com’s February 2026 latency budget analysis puts the architectural implication directly: “Microsecond budgets force data and compute to co-reside. Fetching data from remote storage introduces latency that cannot be hidden.”

Moving compute to the data is faster and cheaper than moving data to the compute. For enterprises with data residency requirements from the EU AI Act, HIPAA, financial services regulations, or sector-specific sovereignty rules, this constraint is also mandatory. The inference cluster must reside in the same jurisdiction as the data. A centralized hyperscaler region that fails the data residency test is not an option regardless of its GPU specifications.

Why Centralized Cloud Cannot Solve AI Inference Latency

Hyperscaler infrastructure was designed for centralization. A handful of major regions handle the majority of compute. This architecture works for batch processing, analytics, and web applications where 200 milliseconds of response time is imperceptible to a human user.

Real-time inference is categorically different. NVIDIA projects inference will be 100 times the scale of training workloads in the near future, per Data Center Knowledge’s December 2025 analysis. That volume of latency-sensitive traffic does not route efficiently through a small number of centralized hubs.

The infrastructure pattern responding to this reality is inference zones: smaller, latency-optimized compute clusters placed close to population centers and data sources. As Iron Mountain’s April 2026 analysis describes, training clusters gravitate toward power-cheap remote locations. Inference clusters follow the user.

A provider operating 200-plus inference locations across geographies places compute in-region for every workload. A provider with 10 to 15 regions cannot. For applications where latency is a functional requirement, the geographic footprint of the infrastructure provider is not a feature comparison point. It is a capability question with a binary answer.

How to Evaluate Infrastructure for Low-Latency AI Inference

Before selecting an inference infrastructure provider, answer four questions in sequence:

1. Where do requests originate? Map the geographic distribution of users, devices, or data sources that generate inference requests. The propagation minimum from each candidate inference location to this origin population is the hard latency floor.

2. Where does retrieval data live? If the inference system retrieves context from a database, document store, or external API, the inference compute needs to reside in the same region. Cross-region retrieval adds propagation cost on every single call.

3. What is the total latency budget? Start from the application requirement. Subtract propagation delay for each candidate region. The remainder is what model inference, tokenization, context retrieval, and application processing can collectively consume. If propagation alone exceeds the total budget, that region does not qualify regardless of its GPU specifications.

4. Are there data residency constraints? If applicable regulations require data to remain within a specific jurisdiction, the compliant region set is determined before any performance analysis begins. Performance optimization operates within the compliance-approved set.

The Connection to Sovereign AI

The Sovereign AI article in this series covered why data residency and regulatory compliance drive AI infrastructure decisions. This article is its performance counterpart.

Regulatory requirements and latency requirements frequently point to the same infrastructure conclusion: compute needs to sit where the data and users are, not where the cloud provider finds it convenient to build. The enterprise that selects geographically distributed infrastructure for compliance reasons typically discovers it has also resolved the latency problems that the centralized model could not.

Both arguments converge on the same architectural principle: inference infrastructure follows the workload. The provider able to place compute in the required region, with 48-hour provisioning speed, enterprise SLAs, and no data residency compromise, satisfies both requirements simultaneously.

About Axe Compute

Axe Compute delivers bare-metal GPU infrastructure across 200+ locations worldwide, provisioned in approximately 48 hours, at up to 80% below hyperscaler rates. No hidden fees. No egress charges. 99.9% uptime.

Sources

 

Recent post