Matilda Bailey has spent years in the trenches of cellular, wireless, and next‑gen networking, guiding teams as AI workloads spill across clouds. In this conversation, she walks through pragmatic ways to collapse deployment from months to days, unify scheduling with Slurm‑on‑Kubernetes, and deliver near‑local data throughput without replication. We dig into direct fiber interconnects with built‑in MACsec, SUNK Anywhere’s control plane, LOTA Cross‑Cloud caching, and deeper Weights & Biases integrations across Google Cloud, AWS, Azure, and CoreWeave. The throughline is simple: remove orchestration silos, bust networking bottlenecks, and neutralize data gravity so researchers can ship faster and Ops can sleep at night.
Which cross‑cloud AI pain points are you tackling first—networking bottlenecks, orchestration silos, or data gravity—and why? Walk us through a real customer scenario, baseline metrics, and the before/after impact on time‑to‑train, deployment lead time, and total cost.
We start where blast radius is highest: the three primary points of failure—networking bottlenecks, orchestration silos, and data gravity—because they compound one another. A recent team spread across CoreWeave and Google Cloud had idle GPUs waiting on data, and separate schedulers that couldn’t coordinate bursts. Baseline, deployments dragged for months while researchers twiddled thumbs; after the direct interconnect and SUNK Anywhere, they were standing up in days and training kicked off as soon as queues opened. Total cost bent downward because they avoided third‑party providers and tapped near‑local data at roughly 7 GB per second per GPU, which cut waste and calmed tempers.
With a direct, fiber‑based interconnect to Google Cloud shrinking deployments from months to days, how do you codify that into a repeatable playbook? Detail provisioning steps, typical lead times, change controls, and the latency/bandwidth SLOs you hold teams accountable to.
Our playbook begins with capacity reservation on the private interconnect, followed by route advertisement and ACL templates that bake in least privilege from hour one. We couple that with profile‑based VLAN and MACsec policy bundles so change requests are diff‑able and auditable rather than bespoke snowflakes. Lead times compress because we skip third‑party carriers; the “months to days” window is unlocked by having fiber ready and configs versioned as code. SLOs focus on sustained near‑local throughput per GPU and predictable low‑jitter paths, and we measure against the 7 GB per second per GPU data target when LOTA Cross‑Cloud is in play.
How does built‑in MACsec reshape your cross‑cloud threat model? Explain key management and rotation, failure handling during rekeys, and audit evidence. Contrast operational overhead and performance with IPsec or TLS overlays, and share any latency deltas you’ve measured.
MACsec shifts us to Layer 2 link protection with hardware assist, so we don’t balloon headers or juggle tunnel sprawl. Keys live in a central KMS, rotation is time‑bound with overlapping windows, and if a rekey stumbles we fail secure and roll back without tearing down the link. Auditors get rotation logs, policy manifests, and packet‑level counters that prove encryption stayed on during change events. Compared with IPsec or TLS overlays, the operational overhead is lighter and the path cleaner; we avoid additional encapsulation while keeping performance aligned with our near‑local data targets.
Where do you see the largest TCO swing after removing third‑party networking providers—transit, idle accelerator time, or engineering toil? Provide a sample cost model, sensitivity analysis by workload size, and common pitfalls that erode expected savings.
The biggest swing is idle accelerator time because months‑long waits shrink to days, and that alone turns sunk cost into throughput. The sample model buckets savings into avoided carrier fees, reduced queueing loss, and fewer handoffs that spawn toil and incident fatigue. Sensitivity rises with dataset size and cross‑cloud hops; the larger the corpus, the more you benefit from near‑local access at roughly 7 GB per second per GPU. Pitfalls include under‑provisioned caching tiers and leaving orchestration silos intact, which quietly hand back your gains.
SUNK Anywhere extends Slurm‑on‑Kubernetes across CoreWeave, Google Cloud, AWS, and Azure. How do you enforce fairness, preemption, and quotas across clusters? Share scheduler configs, queue designs, and how you resolve collisions between research urgency and DevOps reliability.
We mirror Slurm partitions to Kubernetes namespaces and bind them with quotas so a hot research queue can burst, but never swamp platform SLOs. Priority classes map to fair‑share weights, and preemption only kicks when protected DevOps queues hit watermarks. Configs live as CRDs so the same knobs exist across CoreWeave, Google Cloud, AWS, and Azure without drift. When urgency collides with reliability, we open a time‑boxed high‑priority lane with explicit budget and rollback gates, then return to steady‑state after the spike.
What does a unified control plane actually look like—namespaces, CRDs, shared APIs—and how do you manage version drift across clouds? Give a step‑by‑step cutover from single‑cloud to hybrid while keeping long‑running training jobs uninterrupted.
The control plane is a shared API layer where Slurm constructs are expressed as Kubernetes CRDs, and namespaces enforce tenancy and quotas. We freeze versions behind a compatibility matrix and gate upgrades through canaries so one cloud never outruns the others. Cutover starts with read‑only federation, then mirrored queues, then draining and backfilling low‑risk jobs to the second cloud while long‑running training continues undisturbed. Only after health checks go green do we let high‑priority workloads burst across both clouds.
With near‑local throughput around 7 GB/s per GPU via cross‑cloud caching, which workloads benefit most—video, diffusion, retrieval‑augmented training? Explain cache warm‑up, eviction policies, consistency guarantees, and how you detect and prevent cache thrash.
Video and diffusion feel the gain first, but retrieval‑augmented training loves it when hot shards stay resident at roughly 7 GB per second per GPU. Warm‑up starts with manifest‑driven prefetch so we don’t whipsaw the fabric with random pulls. Eviction blends LRU with pinning for golden datasets, and consistency favors read‑mostly workloads with explicit invalidation hooks on write. We watch hit rate, re‑fetch churn, and queue backpressure; if thrash appears, we pin, resize tiers, or reshape IO patterns.
How does a Zero Egress Migration program change data movement economics and governance approvals? Outline a concrete migration timeline, decision checkpoints, and the residual egress or metadata costs that still appear on the bill.
Zero Egress Migration reframes the discussion from “how much will we pay to move” to “how fast can we start using it,” which greases governance wheels. The timeline begins with classification, then cache seeding, then shifting readers, and only later pruning the legacy footprint. Decision checkpoints verify residency, encryption posture, and cache hit rates before broad cutover. You may still see residual metadata and control‑plane calls on the bill, but not the big bulk‑egress shocks.
For validating “near‑local” data access, what benchmarking methodology do you recommend? Specify datasets, IO patterns, tools, and acceptance thresholds. Share common mistakes—cold cache runs, mixed read/write tests, noisy neighbors—and how to avoid them.
Use representative datasets that match shard sizes and access skew, and drive sequential plus strided reads that mirror your loaders. Tools should collect end‑to‑end timing and per‑GPU throughput so you can confirm you’re flirting with the 7 GB per second per GPU neighborhood. Acceptance means sustained throughput under steady concurrency with stable tail latency during warm runs. Avoid cold‑cache measurements, don’t mix heavy writes into read tests, and isolate neighbors so you’re not benchmarking your own interference.
What cross‑cloud failure modes show up most—link saturation, scheduler deadlocks, stale caches—and how do you design runbooks, alerts, and circuit breakers to minimize blast radius? Include MTTR targets and real incident anecdotes.
The common trio is link saturation during bursts, scheduler priority inversions, and caches that go stale when metadata drifts. Our runbooks trigger traffic shaping and queue dampening before starvation sets in, and we pre‑stage fallbacks that keep hot datasets local for a spell. Alerts key off rising re‑fetch rates and growing backlogs so humans step in before jobs wedge. In one memorable incident, we cut over in minutes by pinning a dataset and letting training continue while the catalog reconciled, keeping throughput near the 7 GB per second per GPU norm once healed.
With deeper Weights & Biases integrations—Gemini CLI, Gemma via W&B Inference, TPU utilization telemetry—how do you improve experiment tracking and reproducibility? Walk through a pipeline from prompt to training to inference, including artifacts, metrics, and rollback steps.
We start with Gemini CLI to stamp prompts and configs into versioned artifacts, then wire runs to log hyperparameters, datasets, and environment hashes. During training, TPU utilization telemetry and per‑GPU counters ride alongside loss curves and throughput so drift is obvious. On inference, Gemma via W&B Inference captures model cards and inputs, and artifacts trace back to the exact training run across clouds. Rollback is a click to the last green artifact bundle, restoring weights, data pointers, and the same cross‑cloud path that proved stable.
How do you meet compliance and data residency requirements when caching across jurisdictions? Detail policy controls, encryption in transit/at rest, tenant isolation, and the audit artifacts that satisfy regulators during evidence collection.
We encode residency as policies tied to namespaces, so caches only hydrate within approved regions and never leap borders. Encryption is mandatory in transit and at rest, and built‑in MACsec protects the link while storage uses strong at‑rest keys. Tenant isolation is hard‑walled with network policies and per‑tenant cache segments so no bleed‑through occurs. Auditors receive residency policies, key rotation logs, and cache access records that map directly to job IDs across CoreWeave, Google Cloud, AWS, and Azure.
What portability can teams expect across CoreWeave, Google Cloud, AWS, and Azure? Discuss container images, drivers, and accelerator heterogeneity (GPU vs TPU). Provide a checklist to avoid subtle lock‑in in storage formats, networking, and observability.
Portability is real when container images are hermetic and drivers are pinned through CI so you don’t chase ghosts in new regions. Heterogeneity is addressed by abstracting accelerators behind scheduler profiles, letting GPUs and TPUs slot into the same workflow shape. The checklist: neutral storage formats, network policies that don’t depend on a single‑vendor quirk, and observability that exports standard metrics and traces. Keep seed data portable and configs declarative, and you won’t stumble into accidental stickiness.
Which organizational shifts enable success—roles, OKRs, funding models—and how do you align incentives across platform, security, and research teams? Share an example of reworked on‑call, escalation paths, and incident ownership for cross‑cloud jobs.
We realign OKRs around shared outcomes—time‑to‑train and reliability—so platform, security, and research all win or lose together. Funding follows usage and SLOs, not silos, which motivates teams to reduce queueing and toil. On‑call rotates cross‑functionally, with clear escalation to a unified control plane owner who can nudge Slurm or Kubernetes knobs without turf wars. Incidents name a single commander, a researcher liaison, and a security reviewer, and the postmortem maps fixes to code and policy in days, not months.
How does this approach compare to third‑party network fabrics or managed multi‑cloud schedulers? Offer a trade‑off matrix across time‑to‑value, resiliency, vendor risk, and tooling maturity, and describe when you would still recommend an alternative path.
Direct interconnect and SUNK Anywhere deliver time‑to‑value by chopping deployments from months to days, while keeping resiliency inside your operational model. Vendor risk is spread across clouds without adding another third‑party to the blast radius, and tooling maturity rides on Slurm and Kubernetes you already trust. Third‑party fabrics can look tidy on a slide but often add latency, contracts, and opaque change windows. I’d still recommend them if you lack the skills to run the control plane today or if a niche feature is mission‑critical and not available natively.
What is your forecast for multi‑cloud AI?
The future tilts toward pragmatic interoperability where control planes abstract clouds and data gravity loses leverage. Direct connections, built‑in MACsec, and caching that pushes roughly 7 GB per second per GPU will become table stakes, not headlines. Researchers will expect to start in days, not months, and shift seamlessly between CoreWeave, Google Cloud, AWS, and Azure as capacity ebbs and flows. The winners will be the teams that turn today’s months‑to‑days leap into a habit, codified in playbooks and guarded by shared, cross‑cloud SLOs.
