The Fabric Has to Scale. The NOC Shouldn't.
AI clusters are now measured in tens of thousands of GPUs, and the network that wires them together has to perform perfectly or the job stops. Meta, for example, trained Llama 3 on 16,384 NVIDIA H100s. IREN is standing up more than 50,000 Blackwell Ultra GPUs. Clusters of these sizes only work if the fabric scales to support the needs of the GPUs.
But here’s the problem. The Day-2 operations team can’t grow at the same pace as the fabric, nor should it. There simply aren’t enough people who can debug a 50,000 GPU fabric at 3 a.m., and more alerts won’t fix that. What has to scale is the system that runs the fabric, not the number of people sitting in the critical path.
That gap costs GPU time. In Meta’s 54-day Llama 3 run, there were 419 unexpected interruptions. Thirty-five of them came from a network switch or a connecting cable, and that was enough to stop the job. Someone still has to figure out which warning is real while the GPUs sit idle. That’s the actual cost.
How Day-2 Should Scale
The answer to a fabric that outgrew the team is not more people in the NOC or on the floor. It’s a system that already knows what the fabric should look like, sees when live state drifts, and can put an agent on that picture under policy.
You declare the intent and push it to the fabric. You keep auditing live state against that design and flag drift. One source of truth, not a pile of configs and tribal knowledge.
A GPU job doesn’t care whether the stall came from a switch, a NIC, a GPU, or a storage array. “What’s slow?” should have one place to get asked instead of three consoles and a war room.
Knowing what’s slow still doesn’t close the ticket. That next step belongs to an agent with guardrails: only the tools the operator allows, and a person still signs off before anything changes. Alerts become one incident, the failing path is visible, a fix gets recommended by the agent, a human approves the changes, and the agent confirms the fabric recovered.
That’s the Day-2 model we’re building in Verity.
The Next Step is the Dark Factory
In manufacturing, a dark factory is a plant that can run without a crew on the floor. For fabric operations, it means Day-2 that doesn’t start with a ticket.
That’s the work in front of us:
- Closed-loop RCA – Investigate, propose the fix, and apply it inside a policy boundary instead of handing the last step to someone else.
- Predict fabric failures before a job stalls – Catch drift while the GPUs are still working, not after they’re already waiting.
- Self-heal routine faults – A flapping optic or a bad link gets handled without a ticket ever being opened.
That’s the path: take the routine work off the ticket queue, under policy, one verified step at a time.
그렉 존스
최고 데이터 책임자
Greg는 의료, 생명공학, 금융, 주 및 연방 정부, 전자상거래, 제조 등 다양한 산업 분야에서 20년 이상 대기업 개발을 설계하고 이끈 경력을 보유하고 있습니다. 네브라스카 대학교에서 전기 공학 학위를 받았으며 매사추세츠 공과대학에서 머신 러닝 교육을 이수했습니다.
