Matilda Bailey is a heavy hitter in the networking world, bringing years of expertise in cellular, wireless, and next-generation solutions to the table. As AI infrastructure becomes the backbone of the modern enterprise, her unique perspective on how hardware metrics intersect with real-world performance has become an essential roadmap for organizations trying to scale. We sat down with her to discuss the intricate dance of managing high-performance compute environments, where the difference between a successful model and a costly failure is often measured in milliseconds and gigabytes.
Our conversation dives deep into the four pillars of AI hardware management: compute, networking, storage, and business logic. We explore the hazards of GPU memory saturation, the necessity of high-speed fabrics like 800 GbE, and why storage throughput is the true lifeline of a modern data lake. Matilda also breaks down the essential observability tools—ranging from open-source favorites to specialized vendor solutions—that turn raw telemetry into actionable insights for IT leaders.
GPU memory saturation often forces data onto slower system RAM, causing performance to tank. How do you advise teams to balance high-speed VRAM usage without hitting those critical out-of-memory errors?
When you are pushing these high-performance mathematical operations, you really have to treat GPU memory like the most expensive real estate on earth. GPU memory saturation is a silent killer for training runs; the moment your video random access memory, or VRAM, is fully tapped out, the system tries to compensate by offloading data to the system RAM. The problem is that this movement happens across the PCIe bus, which is a fundamentally suboptimal interface for the sheer volume of data AI models require. You’ll feel the performance degradation immediately as the training cycles crawl to a halt or, worse, you start seeing those dreaded out-of-memory errors that can scrap hours of progress.
To stay in the safe zone, I generally recommend that teams keep their utilization between 60% and 80%. This provides a vital buffer for model parameters, gradients, and other data elements that can fluctuate during a run. If you find your team consistently hitting that ceiling, it’s a clear signal that your current hardware—which in 2026 might range from 16 GB to 140 GB per GPU—is simply not enough for the workload. You have to monitor that capacity alongside saturation metrics to justify whether you need to invest in more robust units or if you can optimize the model to fit within the existing footprint.
Networking is often the unsung hero of AI, yet a single dropped packet can stall a massive cluster. What specific network metrics should engineers prioritize to ensure seamless data flow across high-speed fabrics?
In the world of AI, networking is the vital interconnectivity that keeps the whole organism alive; without it, even the fastest processors are just idle silicon. We are now seeing environments that depend on massive throughput like 400 Gigabit Ethernet or even 800 GbE and InfiniBand to maintain the pace. The metric I tell everyone to watch like a hawk is the AI fabric packet loss. While a few dropped packets are annoying in a standard business network, in an AI fabric, they are catastrophic because the time spent resending that data impairs the entire training performance. You want to aim for a zero-loss network architecture to maintain an optimum Model Floating-point Operations per Second, or MFU.
Beyond packet loss, you really need to keep an eye on interGPU latency and RDMA or RoCE drops. Remote Direct Memory Access is incredible because it allows systems to exchange data without involving the CPU, which vastly improves performance, but when you start seeing drops in those transfers, you’re looking at serious synchronization problems. It’s also important to monitor queue depths and buffer use; if those buffers are filling up, it’s a physical manifestation of congestion. It means your data is literally sitting in a waiting room while your GPUs are starving for information, wasting both time and massive capital investment.
Compute efficiency isn’t just about raw speed; it’s about the ratio of theoretical performance to actual output. Why is a metric like Model Floating-point Operations per Second utilization so telling for AI infrastructure health?
MFU is essentially the “truth teller” of AI infrastructure because it measures the ratio of the compute performance you’re actually achieving against the peak hardware capacity of your GPUs or TPUs. It’s common for people to see a high GPU utilization rate—say, 90% or 95%—and assume everything is perfect. But utilization only tells you that the processor is active; it doesn’t tell you if it’s doing useful work. If your GPU is active 95% of the time but your MFU is basement-level low, you have a massive efficiency gap, likely caused by unoptimized code or data bottlenecks.
In actual practice, hitting 100% MFU is a fantasy, but if you can achieve 40% or better, you are generally looking at a healthy, well-performing model. This metric allows you to see if the billions of floating-point operations per second you paid for are actually translating into training progress. When MFU drops, it’s often a sign that you need to look back at your memory bandwidth or your interconnects. If the data can’t feed the GPUs fast enough, they will pause and wait for the VRAM to catch up, which drives down your MFU and makes your entire operation less cost-effective.
As data lakes grow into the petabyte range, storage subsystems face unprecedented pressure. How does storage latency specifically impact the efficiency of model weight updates during training?
Storage is the foundation of the entire AI pipeline, and as we move from the terabyte range into the petabyte range, the physical distance between the data and the GPU becomes a major hurdle. Storage latency is the time it takes to move that data from your data lake into the GPU memory, and even a slight increase can ripple through your entire training schedule. High latency slows down weight updates and delays the transfer of fresh data coming in from IoT or other real-world sources. It’s a frustrating experience for engineers who have optimized their models only to be throttled by a slow storage read.
To combat this, we monitor storage throughput in gigabytes per second to ensure the “pipes” are wide enough to keep the processing pipelines filled. But you also have to look at IOPS, or Input/Output Operations Per Second, especially for metadata-centric workloads or random inference queries. If you see a disruption in IOPS, it’s an early warning sign of a storage bottleneck. Maintaining a low-latency, high-throughput storage environment is the only way to ensure that your processors aren’t sitting idle while the storage subsystem struggles to find and deliver the next block of data.
With a suite of tools ranging from open-source toolkits like Prometheus to specialized hardware managers like NVIDIA’s Base Command, how should an organization build its observability stack?
Building an observability stack is about finding the right balance between broad visibility and deep, specialized insights. For the core infrastructure, open-source tools like Prometheus are fantastic for cloud-native environments because they are highly flexible for collecting and reporting metrics. However, for the more granular, hardware-specific data, you really need tools like the NVIDIA DCGM Exporter or the PyTorch Profiler. These allow you to see execution time, GPU kernel utilization, and even memory consumption during specific training runs, which is something a general monitoring tool might miss.
I also see a lot of success with unified platforms like Datadog, Grafana Cloud, or New Relic. These platforms are great because they correlate model behavior with traditional system metrics, giving you a “single pane of glass” view. For instance, you can see a spike in ML model latency and immediately cross-reference it with network packet loss or storage IOPS in the same dashboard. Whether you use Zabbix for server health or Coralogix for security and logs, the goal is to ensure that your hardware metrics aren’t siloed away from your model performance data. You need that correlation to truly optimize your costs and prevent bottlenecks before they crash a project.
What is your forecast for AI infrastructure development over the next few years?
My forecast for AI infrastructure is a shift toward “zero-friction” hardware, where the boundaries between compute, storage, and networking become almost invisible through tighter integration. We are already seeing this with the move toward 800 GbE and the adoption of specialized neural processing units, but the real change will be in how we manage these resources. I expect to see AI observability platforms become even more “agentic,” where the system doesn’t just alert you to a bottleneck but automatically reconfigures the network fabric or shifts workloads to different GPU clusters to maintain that 40% MFU or higher. As GPU memory capacities continue to climb toward and beyond that 140 GB mark, the challenge won’t just be having enough power, but orchestrating it efficiently enough to keep the massive costs of this infrastructure under control. In 2026 and beyond, the winners will be those who master the metrics of efficiency, not just those with the biggest hardware budgets.
