AI Agents: The Runtime Infrastructure Problem
The model is rarely the culprit. Three surveys from spring and summer 2026 all point at the same place, the operations layer your agents run on. That's where projects live or die.
The short version: 37 percent of 132 technology leaders surveyed in May 2026 blame the runtime layer for stalled agent projects, just 17 percent blame the model. Which matches what LinkedIn told the VB Transform audience. Their agents first choked on container start-ups in the seconds range, and the fix was unsexy: keep pre-provisioned pools warm, build your own control flow, push the language models to the leaf. Around 80 percent of the workflow there is now plain scripted code. One caution, though. A second survey found 71 percent of companies conceding that at most a quarter of their so-called agents do any multi-step orchestration at all.
The problem sits in the runtime
Don't go looking for the fault in the model. When 132 technology leaders were asked in May 2026 why their agent pilots were stuck, the runtime layer topped the list of primary failure modes. The supposedly-too-dumb model? Far down the list.
Which turns the usual debate on its head. Waiting for the next model update buys you little while nobody can answer what your agents actually run on.
The reason is mundane, almost architectural. Web infrastructure answers short, stateless requests, that's what it was built for. An agent doesn't play along. It runs for minutes, sometimes hours, drags intermediate state around and knocks on external systems dozens of times. A foundation poured for web shops won't carry that.
The models are smart enough, but our stateless infrastructure is too fragile to manage long-running, multi-step agentic processes.
What follows is manual work. 77 percent of respondents sink real engineering capacity into infrastructure plumbing, and for 24 percent it eats more than half the sprint. Time no feature ever sees.
Where the friction really comes from
Latency isn't the main problem. The headlines say otherwise, the data doesn't: in the same survey, latency jitter drops from 25 to 11 percent as a maturity criterion. User acceptance and context fidelity sit on top instead. Put differently, the systems are fast by now. Reliable across long runs they are not.
Production-readiness metrics named by respondents, VentureBeat Pulse Research, May 2026, n=132.
One floor down, in the data centre, it sounds different. There, 71 percent of IT decision makers report latency problems with AI workloads, against 32 percent in 2024. Asked for the biggest barrier to expansion, 40 percent say IT infrastructure. Budget? 1 percent.
Before you quote that number onwards: it comes from a colocation provider, someone who earns money when infrastructure counts as the bottleneck. Take it as a direction. Nothing more.
And cost? EY runs the numbers: a customer service interaction that went for about 0.04 US dollars in 2023 lands at roughly 1.20 US dollars in 2026 as an orchestrated agent run. An illustration, not a measured average. The direction holds all the same, every notch of autonomy gets paid for in tokens. How to keep that in check is covered at length in Token FinOps .
How LinkedIn solved it
At VB Transform 2026, LinkedIn, Walmart and Zendesk talked about what their agents ran into first. At LinkedIn: Kubernetes. The platform assumes a container may take a second or two to come up. An agent calling tools on a millisecond cadence disagrees.
The first answer sounds almost disappointing: LinkedIn stopped spinning containers up on demand and keeps pre-provisioned pools warm instead. That's it.
The second step weighs more. Animesh Singh describes it as: own harness, own control flow, and the language models move to the leaf instead of the middle of the loop. Around 80 percent of the workflow there is scripted, deterministic code today. The model only gets called where language understanding is genuinely needed.
We built our own harness, our own control flow, and pushed the LLMs to the leaf instead of them orchestrating the loop.
Animesh Singh, LinkedIn, at VB Transform 2026Zendesk handles 20 billion customer conversations by its own count. Sami Ghoche says the common denominator of systems that last is evaluations. And Desiree Gosby of Walmart frames the goal strikingly defensively: don't slide back into the state where engineering is the bottleneck.
The lesson cuts against the marketing: less autonomy at the core, more fixed structure around it. If you want the long version, we wrote it up in Harness Engineering .
Most agents are not agents
An honest look at your existing stock pays off before money goes into an orchestration platform. In June 2026, 101 companies were asked what their so-called agents actually do. The result: most of it is chatbot wrappers with a better name. The platform layer gets built anyway, just ahead of the portfolio it's supposed to run.
| Share of genuinely orchestrated agents | Companies |
|---|---|
| 0 percent | 9 percent |
| 1 to 25 percent | 62 percent |
| 26 to 50 percent | 19 percent |
| 51 to 75 percent | 7 percent |
| 76 to 100 percent | 3 percent |
Taken together, 71 percent concede that at most a quarter of their agents do genuine multi-step orchestration. Just 10 percent get past halfway. The survey authors describe an orchestration layer that keeps running ahead of the portfolio it is meant to steer.
On platforms: Anthropic Claude leads at 40 percent, Microsoft follows at 18, OpenAI at 13. What interests us more is a different number. 27 percent can't stop a runaway agent in real time, they find out from the invoice. That's not a model question. That's operations.
The research community is similarly sober. An ICML paper from February 2026 tested 15 models across twelve metrics, and reliability barely moved despite all the capability gains. Waiting for the next model solves the wrong problem.
The European perspective
For European companies there's one more layer on top: location. Agents in production need predictable compute capacity. And that hangs on two rather mundane things, grid connection and electricity prices.
Bitkom and Borderstep supply the numbers: 530 megawatts of AI capacity in German data centres in 2025, an expected 2,020 megawatts by 2030. Total capacity stood at 2,980 megawatts in 2025 and crosses the 3,000 mark in early 2026. Sounds like momentum. There's a catch though, approval procedures run about six months beyond the statutory timeframe according to Bitkom, longer than the EU average.
Adoption, meanwhile, is pulling ahead fast: from 17 to 41 percent of German companies within a year, per the Bitkom survey from March 2026. Asked about barriers, 77 percent answer data protection, 70 percent skills shortage, 61 percent technical security.
Which leaves sovereignty. Three US hyperscalers hold roughly 65 percent of the European cloud market, says Gaia-X. The EU Cloud and AI Development Act therefore wants to triple data centre capacity within five to seven years. Whether that's enough is another matter.
To be straight about it: there is no solid survey yet on agent runtime infrastructure specifically in Europe. Tying the sovereignty debate to agent operations is an argument, not a measurement. Anyone selling it as a number in a board paper is overreaching.
Challenges and risks
Before you budget against the runtime thesis, know its limits. Part of the evidence comes from vendors who profit from the answer. The samples are small. And a surprising number of circulating agent statistics can't be traced to any primary source at all.
And then there's time. Rebuilding the runtime layer ties up months of engineering in something the business side never lays eyes on. The only way to sell that internally is with a number in hand: how much sprint time already leaks into repairs.
What companies should do now
What follows from all this? Nothing expensive. Before an orchestration platform lands on the shopping list, two questions deserve answers: is there anything to orchestrate at all? And does the runtime environment survive long-running processes?
Six steps, in this order
-
Count what you have
How many of your agents genuinely work in multiple steps? Count them. If little comes back, the platform question simply isn't due yet, a chatbot with tool access doesn't need an orchestration layer.
-
Measure start-up time
Time the start-up of your execution environment. If a container takes seconds, that throttles every agent workload. Warm pools help, and cost base load in return.
-
Cut autonomy
If a sequence is fixed, it belongs in code, not in the model. That's exactly how LinkedIn gets to its roughly 80 percent scripted share. Autonomy is not an end in itself.
-
Build a kill switch
Build a kill switch that stops running agents on the spot and enforces a spend limit. 27 percent of surveyed companies still can't do this. They find out from the invoice.
-
Evaluations before scale
Evaluations come before scale. They show up at every company that keeps agents in production for the long haul. That's no coincidence.
-
Settle the location question early
In Europe, grid connection, electricity price and approval timelines co-decide where your agent operations can grow. Settle that at architecture time. Later it gets expensive.
More important than any tool is the order. Start at step one and the platform debate often postpones itself for another year.
Further reading
Frequently asked questions
The engine room beneath the model and the framework. Concretely: the execution environment with its start-up time, state across long runs, connections to external systems, and control over cost and abort. Sounds unglamorous, decides whether an agent survives operations.
Web infrastructure was built for short, stateless requests. An agent does the opposite, runs minutes to hours, holds intermediate state, knocks on other systems dozens of times. The May 2026 survey of 132 technology leaders confirms the mismatch: 37 percent see the runtime as the primary failure mode, 17 percent the model.
Containers first: Kubernetes budgets start-up in the seconds range, so LinkedIn now keeps its pools warm ahead of time. Then the control flow: built in-house, with the language models at the leaf rather than mid-loop. The result is roughly 80 percent of the workflow as deterministic scripted code.
No. Latency jitter fell from 25 to 11 percent as a production-readiness metric (May 2026). What worries respondents instead: user acceptance at 47 and context fidelity at 30 percent. Speed the systems can do. Staying reliable over long runs they can't, yet.
Fewer than the word suggests. In the June 2026 survey of 101 companies, 71 percent said at most a quarter of their agents genuinely orchestrate in multiple steps. Nine percent admitted: none. What's left are chatbots with a tool belt.
Start with a stocktake. Count how many agents genuinely run multi-step, and time how long your execution environment needs to start. After that: kill switch with a spend limit, evaluations, and only once all that stands, scale. The platform question you save for last.