Matilda Bailey has spent years at the intersection of high-performance networking and next-gen infrastructure, witnessing the transition from static cloud services to the frantic demands of modern artificial intelligence. As enterprises move beyond simple chatbots toward complex agentic systems, she has observed a critical friction point where current hardware designs struggle to handle the complex orchestration required by these autonomous models. Her work focuses on bridging the gap between the raw power of the GPU and the sophisticated logic of the host CPU to create a more cohesive computing environment.
This discussion explores how the transition to agentic AI is exposing significant flaws in traditional, GPU-heavy server environments. The conversation covers findings from production telemetry that reveal a “ping-pong” effect between processors, the inefficiencies of static resource provisioning that leave hardware underutilized, and the emergence of “workflow-aware” architectures like Agora. Ultimately, the focus shifts from individual components to a holistic view where the entire server, rack, and runtime operate as a single, heterogeneous system to meet the demands of multi-step AI workflows.
The industry is shifting from basic chatbots to complex agentic AI, but recent research suggests our current server designs might not be up to the task. What is happening under the hood when these agents execute a task?
When we look at how agentic AI functions, we see it isn’t just a bigger version of the chatbots we’ve grown used to over the last few years. Instead of a single, linear request, these agents operate as distributed applications that execute dynamic, multi-step workflows. In a controlled study using the CORAL framework, researchers watched a single workload explode into 580 LLM calls interleaved with 552 tool invocations. This creates a frantic “ping-pong” effect where the execution jumps back and forth between the CPU and the GPU hundreds of times. It’s a sensory overload for the hardware; the system is constantly context-switching, which exposes massive inefficiencies in how we’ve traditionally built our AI infrastructure.
You mentioned this “ping-pong” effect between processors. How does this constant switching impact the actual utilization of the hardware we are spending millions on?
The reality is that our current static resource provisioning is becoming a major liability because it can’t keep up with these fragmented execution patterns. We see host CPU utilization remaining low for long stretches, only to spike sharply during bursts of tool execution, while GPUs might sit idle waiting for the next instruction. This creates “transient bottlenecks” where either processor can suddenly become the sticking point on a workflow’s critical path. Interestingly, tool time actually matched or even beat inference time in more than 27 percent of requests. This means that if you are only looking at average utilization numbers, they are essentially lying to your infrastructure teams about the health and efficiency of your system.
If the traditional GPU-centric model is failing to handle these agentic workflows, what does a “workflow-aware” server architecture actually look like in practice?
A new architecture called Agora offers a compelling blueprint by treating the server as a flexible pool of resources rather than a collection of silos. It works by dynamically harvesting idle CPU cores for throughput work while protecting the system against the sudden latency spikes that occur during tool execution. To solve the memory bottleneck, Agora oversubscribes GPU memory by placing more agents on each GPU and using state prefetching to hide the time it takes to swap data. This approach moves away from a one-size-fits-all model and instead pools cores by specific roles—like scheduling or orchestration—and applies affinity-aware scheduling. When you see this in action, the results are staggering: it can increase host CPU utilization by about 30 percent and free up roughly one-third of the GPUs through smarter consolidation.
How should enterprise leaders rethink their procurement strategies if the individual processor is no longer the sole king of the data center?
We have to accept that the “throne” of the individual processor is disappearing, and the competitive advantage is moving to how the entire rack and runtime operate as one cohesive system. Instead of just buying the fastest “box” or the most expensive GPU, organizations need to focus on procuring the workflow itself. The data shows that workflow-aware designs can increase generation throughput by 82 percent while reducing tail latency by 2.5 times, which is a massive leap in performance. Leaders should prioritize workload profiling and sophisticated scheduling because these elements will deliver far greater benefits than simply sizing infrastructure based on model inference alone. It’s about the choreography of the entire system, not just the strength of a single dancer.
What is your forecast for the future of enterprise AI infrastructure?
I believe we are entering an era of “heterogeneous orchestration” where the distinction between the CPU and GPU roles will become increasingly blurred in favor of specialized, role-based processing. Within the next few years, the standard for a “high-performance” server will be defined by its ability to recover 95 percent of a co-located workload’s throughput under low load rather than its peak theoretical FLOPs. We will see a massive shift toward software-defined hardware that can adapt in real-time to the “ping-pong” nature of agentic workflows. Ultimately, the winners in this space won’t be those with the most hardware, but those who can most efficiently manage the complex, fragmented execution graphs that define the next generation of artificial intelligence.
