The Fabric Has to Scale. The NOC Shouldn't.
AI clusters are now measured in tens of thousands of GPUs, and the network that wires them together has to perform perfectly or the job stops. Meta, for example, trained Llama 3 on 16,384 NVIDIA H100s. IREN is standing up more than 50,000 Blackwell Ultra GPUs. Clusters of these sizes only work if the fabric scales to support the needs of the GPUs.
But here’s the problem. The Day-2 operations team can’t grow at the same pace as the fabric, nor should it. There simply aren’t enough people who can debug a 50,000 GPU fabric at 3 a.m., and more alerts won’t fix that. What has to scale is the system that runs the fabric, not the number of people sitting in the critical path.
That gap costs GPU time. In Meta’s 54-day Llama 3 run, there were 419 unexpected interruptions. Thirty-five of them came from a network switch or a connecting cable, and that was enough to stop the job. Someone still has to figure out which warning is real while the GPUs sit idle. That’s the actual cost.
How Day-2 Should Scale
The answer to a fabric that outgrew the team is not more people in the NOC or on the floor. It’s a system that already knows what the fabric should look like, sees when live state drifts, and can put an agent on that picture under policy.
You declare the intent and push it to the fabric. You keep auditing live state against that design and flag drift. One source of truth, not a pile of configs and tribal knowledge.
A GPU job doesn’t care whether the stall came from a switch, a NIC, a GPU, or a storage array. “What’s slow?” should have one place to get asked instead of three consoles and a war room.
Knowing what’s slow still doesn’t close the ticket. That next step belongs to an agent with guardrails: only the tools the operator allows, and a person still signs off before anything changes. Alerts become one incident, the failing path is visible, a fix gets recommended by the agent, a human approves the changes, and the agent confirms the fabric recovered.
That’s the Day-2 model we’re building in Verity.
The Next Step is the Dark Factory
In manufacturing, a dark factory is a plant that can run without a crew on the floor. For fabric operations, it means Day-2 that doesn’t start with a ticket.
That’s the work in front of us:
- Closed-loop RCA – Investigate, propose the fix, and apply it inside a policy boundary instead of handing the last step to someone else.
- Predict fabric failures before a job stalls – Catch drift while the GPUs are still working, not after they’re already waiting.
- Self-heal routine faults – A flapping optic or a bad link gets handled without a ticket ever being opened.
That’s the path: take the routine work off the ticket queue, under policy, one verified step at a time.
Greg Jones
Chief Data Officer
Greg has more than 20 years of experience designing and leading large enterprise developments for various industries including healthcare, biotech, finance, state and federal government, ecommerce, and manufacturing. He holds an Electrical Engineering degree from the University of Nebraska and completed machine learning training at Massachusetts Institute of Technology.