FarmGPU retweeted
The NVIDIA Vera Rubin NVL72 scale-up fabric, built on the sixth-generation NVIDIA NVLink.
The NVLink 6 Switch trays connect 72 Rubin GPUs in an all-to-all configuration.
The NVLink spine—over 5,000 copper cables make up the vertical NVLink traverse. It securely moves more data than the entire bandwidth of the internet.
The single-wide NVIDIA MGX rack design unlocks a cable-free, horizontal NVLink scale-up traverse, running directly on PCB for resiliency at scale.
This is how NVIDIA Vera Rubin NVL72 works as one, designed to deliver the lowest token cost and 10x more performance per watt than the previous generation.
#NVIDIAVeraRubin
Training backend = rail-optimized collectives.
Inference backend = lossless RDMA for P/D disaggregation, KV cache, isolation.
We run OCP Inference Fabric RA as our backend fabric: DS5000 + SONiC + @githedgehog declarative GitOps.
@Celestica_Inc opencompute.org/documents/op…
OCP NVMe SMART telemetry is richer than most GPU telemetry — we just don't use it.
Most GPU clouds monitor GPU temp, GPU util, and call it observability.
The OCP Datacenter NVMe spec exposes 30+ telemetry dimensions per drive that most fleets never collect: NAND wear curves, ECC escalation rates, recoverable error histograms, thermal throttling counters, grown defect list growth, power-state transition latency.
We built a fleet-wide predictive failure engine on top of that surface. Every drive in 3 sites scored for degradation trends. Replace / Watch / Monitor — 3 weeks before failure, not after.
For clarity: base NVMe SMART / Health Log is foundational. % used, media & data integrity errors, composite temp, unsafe shutdowns, error log depth. If you're not alerting per-drive per-hour on it, you're flying blind. We do.
OCP doesn't replace SMART — it extends it. Endurance group histograms, physical media error bucketization, thermal history, PLP event counters, grown defect trending. SMART = what *is* wrong. OCP = where it's *going*.
That's where 10+ years of OCP + NVMe storage work matters. We run both layers together: SMART for immediate health, OCP telemetry for slope. That's how you go from reactive RMA to scheduled replacement. Storage isn't secondary to GPU reliability — it *is* GPU reliability. 🌾
OCP survey of 19 neocloud operators: top pain points are RoCE tuning, firmware management, and BMC/BIOS inconsistency.
Translation: every neocloud under 500 racks is independently re-discovering the same failure modes and filing the same tickets to the same 3 OEMs.
This is the neocloud ops tax. Not CapEx. Not power. It's the $2M/yr in hidden engineering time spent building bespoke workarounds for Redfish endpoints that work on Dell and break everywhere else. For NIC firmware that needs per-rail tuning. For SONiC versions that handle ECN differently.
FarmGPU + Hedgehog + Celestica built a reference fabric specifically to kill this tax: Celestica DS5000 (51.2T) + SONiC + Hedgehog K8s-native control. 36 composition variants, declarative, vendor-neutral. Contributed to OCP so other neoclouds don't have to reinvent it.
The whole thesis: write down what neoclouds actually need -> get OEMs to ship it through OCP -> open-source the operator tooling that makes the spec real.
Neoclouds don't need another GPU SKU. They need to stop paying that ops tax.
blog.farmgpu.com/ocp-scaling… 🌾🚜
Meta contracted 5+ GW of data center capacity in H1 2026 alone — and nobody's building enough.
Now they're doing "SpaceX-style" bulk compute leasing. New market segment, much higher revenue per megawatt than traditional cloud.
The compute shortage isn't ending. Dylan Patel projects OpenAI + Anthropic could demand 100+ GW by 2030.
FarmGPU's capacity is the asset. The cloud platform is just the interface.
newsletter.semianalysis.com/…
SK Hynix just removed the price cap from its long-term memory contracts.
On its own, a contract structure change. In context, a signal about who holds pricing power in the AI stack.
For years, memory LTAs had ceilings — when spot prices spiked, contract buyers were protected. Micron still does this. SK Hynix is now saying: no ceiling. If the market goes higher, you pay more.
With DDR5 retail having roughly tripled since October, there's a lot of upside to be exposed to.
This splits the memory market into two regimes. Micron customers get predictability at very high Q2'26 levels. SK Hynix customers get uncapped exposure. And SK Hynix is the dominant HBM3E supplier for NVIDIA — so the downstream repricing runs through SSD controllers (which need DDR5 buffers), through eSSD procurement, through GPU cloud TCO, all the way to inference cost per token.
The deeper signal: SK Hynix believes this shortage lasts long enough that uncapped contracts beat capped ones. They're willing to risk customer relationships on it.
When the dominant HBM supplier for the AI accelerator market decides pricing power is theirs to keep, every layer downstream reprices. The memory shortage isn't a cycle — it's becoming the architecture of AI cost.
OCP just released the AI Inference Fabric Reference Architecture — the companion to the Training Fabric RA we co-developed with Hedgehog and Celestica.
It's built on the exact stack FarmGPU runs in production:
Celestica DS5000 (51.2T) + SONiC + Hedgehog K8s-native control plane.
Inference ≠ Training. Different fabric demands:
→ North-south throughput for model serving
→ Multi-tenant isolation
→ Gateway services for ingress/egress
→ Lossless RDMA across hosts for distributed inference
64–1024 xPU clusters. Open, vendor-neutral, declarative. Production-ready blueprints instead of slideware.
This is how neoclouds stop reinventing the network for every deployment.
opencompute.org/documents/op…
🌾🚜
Shepherd: 51 AI agents watching your cluster so you don't have to.
Every farm runs out of time, eyes, and bandwidth. Shepherd agents never do.
51+ autonomous agents across 8 domain crews — power, storage, network, security — operating on a 5-minute loop. CrewAI orchestrator. Fine-tuned triage model that classifies incidents before you wake up.
No dashboards. No alerts you have to manually check. Just agents that fix things, escalate what matters, and keep your cluster alive at 3 AM.
blog.farmgpu.com/b200-cluste…
🌾🚜
OCP just released the AI Inference Fabric Reference Architecture — the companion to the Training Fabric RA we co-developed with Hedgehog and Celestica.
It's built on the exact stack FarmGPU runs in production:
Celestica DS5000 (51.2T) + SONiC + Hedgehog K8s-native control plane.
Inference ≠ Training. Different fabric demands:
→ North-south throughput for model serving
→ Multi-tenant isolation
→ Gateway services for ingress/egress
→ Lossless RDMA across hosts for distributed inference
64–1024 xPU clusters. Open, vendor-neutral, declarative. Production-ready blueprints instead of slideware.
This is how neoclouds stop reinventing the network for every deployment.
opencompute.org/documents/op…
🌾🚜
OCP just released the AI Inference Fabric Reference Architecture — the companion to the Training Fabric RA we co-developed with Hedgehog and Celestica.
It's built on the exact stack FarmGPU runs in production:
Celestica DS5000 (51.2T) + SONiC + Hedgehog K8s-native control plane.
Inference ≠ Training. Different fabric demands:
- North-south throughput for model serving
- Multi-tenant isolation
- Gateway services for ingress/egress
- Lossless RDMA across hosts for distributed inference
64–1024 xPU clusters. Open, vendor-neutral, declarative. Production-ready blueprints instead of slideware.
This is how neoclouds stop reinventing the network for every deployment.
opencompute.org/documents/op…
Leopold Aschenbrenner’s Situational Awareness LP just dropped its Q1 2026 13F, and the move that stands out isn’t the longs — it’s the puts. The fund layered on multi-billion dollar bearish positions across the semiconductor stack: more than billion notional on SMH, over .5 billion on NVDA, roughly billion on AVGO, plus sizable books on AMD, TSM, ASML, MU, and fresh INTC puts while closing prior long calls. Last quarter they held basically zero puts. This is not protective hedging. This is a deliberate, sized short on the most crowded, highest-multiple corner of the AI hardware trade.
The deeper context is Aschenbrenner’s core thesis from his 2024 essay “Situational Awareness: The Decade Ahead.” He puts AGI around 2027 and superintelligence shortly after, but argues the binding constraint is physical infrastructure, not intelligence. Trillions in capital expenditure will be required for clusters, power generation, chips, cooling, and data centers. The fund’s edge is supposed to come from seeing where the real bottlenecks sit before traditional markets price them in.
What the filing shows is a tactical rotation inside that same long-term view. Aschenbrenner is staying aggressively long the broad AGI buildout while exiting or hedging the part of the supply chain that got the frothiest: semiconductors. At the same time he is adding heavily to power, energy, and flexible hosting plays — CleanSpark added over 10 million shares, CoreWeave another million-plus, with continued exposure to Riot, IREN, Applied Digital, and Bloom Energy. The signal is clear: semis are priced for perfection, while power and rapid-deploy compute capacity remain the scarce resources that will actually determine who captures margin as the buildout continues.
Bitcoin miners that have quietly converted capacity to AI hosting now look especially well positioned under this lens. They already cleared the hardest hurdle — securing grid interconnection and power. That advantage is suddenly worth far more than a slightly better chip process node. This filing is sophisticated capital acknowledging that the game has moved downstream. The people who wrote the AGI timelines are now positioning for a world where power is the new oil and hosting flexibility is the real moat.
Ohh hi there @LebanonJon
Big moment for the future of AI in our region. 🚀
Today, Sacramento State joined industry partners, education, and civic leaders in signing a Memorandum of Understanding (MOU) to formalize a first-of-its-kind AI workforce pipeline from kindergarten through career. 📚 💼
The AI K-Career initiative brings schools and industry partners together to align on what education, skills, and training are needed to connect people to jobs in the @CityofRCordova—setting a new standard for how regions can prepare talent for the future.
"This is exactly the kind of collaboration that defines what a university like ours does. We prepare students to create a better life through alignment with industry partners, ensuring they graduate with gainful employment. We're glad to be in partnership with all the amazing people around this table and looking forward to being a core part of what is going to continue to transform this region." - President @DrLukeWood
@flcfalcons
@FolsomCordova
@FarmGPU
@hmci_ai
@NVIDIAAI
@SacStateCarlsen
@solidigm
📸 Sacramento State/Andrea Price
ALT A group of individuals wearing business attire and caps stands in a conference room, holding a large sign with various company logos.
ALT A group of people are seated at a long conference table in a bright room with large windows. Various hats and papers are arranged in front of the individuals, who appear to be engaged in a discussion.
ALT Jonmichael Hands, Luke Wood, and Dr. Art Pimentel are seated at a conference room table. President Wood is signing a document.
ALT A display board titled "AI Nucleus: K-Career Signing Ceremony" with logos of Rancho Cordova, NVIDIA, Human Machine Collaboration Institute, Sacramento State, Folsom Lake College, Solidigm, FarmGPU, and SMUD.
The neoclouds quietly became the second tier of hyperscale AI infra. Same 100kW racks, same RoCE fabrics, same firmware nightmares — on a fraction of the team and balance sheet. That's the gap OCP's new Scaling AI Clusters at Neoclouds workstream exists to close, and FarmGPU is co-chairing it alongside Scaleway and Denvr Dataworks.
A few signals from the first 19-operator community survey worth paying attention to:
→ Power sourcing and benchmarking are rated the most important 2026 problems (4.21 / 4.05 importance).
→ But the area where operators think OCP can move the needle most is open rack and pod reference patterns for 30–100kW (4.11 leverage). Hardware ≠ the bottleneck. Operational surface area is.
→ The top operator pain points: networking (RoCE/IB tuning, congestion control), management & automation (Day-2 ops, firmware), and interoperability (BMC/BIOS/telemetry consistency). Every neocloud is independently re-discovering the same gaps and filing the same tickets to the same OEMs.
Concrete output already in motion:
1. A neocloud Redfish profile. Redfish works great on Dell, inconsistently everywhere else. Instead of 20 operators each begging OEMs separately, the workstream is drafting a collective, machine-validatable profile that codifies the BMC, firmware, TLS, telemetry, and event-subscription behaviors neoclouds actually need. One spec OEMs implement against. One checklist procurement validates.
2. FarmGPU is the first neocloud running an OCP-aligned backend fabric in production — the AI Training Fabric Reference Architecture we co-developed with Hedgehog and Celestica. Celestica DS5000 (51.2T) + SONiC + Hedgehog as the K8s-native control plane. 36 ready-to-deploy composition variants spanning OPG-64/128/256/512 → XOC-256/512/1024, air- and liquid-cooled, single- and dual-plane. Vendor-neutral, declarative, reproducible. Contributed back to OCP so nobody has to reinvent it.
3. OCP NVMe SSD spec → predictive failure engine. We standardize on the OCP Datacenter NVMe SSD spec because the telemetry surface is dramatically richer than vanilla NVMe. On top of that we built a fleet-wide predictive failure engine — scores every drive for NAND degradation, ECC escalation, recoverable/uncoverable error trends, surfaces replace/watch/monitor recommendations. Plan is to open-source it so other neoclouds can run it directly.
The whole thesis: write down what neoclouds actually need → get OEMs to ship it through OCP → open-source the operator tooling that turns the spec into operational value. That compounds.
Full writeup, survey results, hardware-management deep dive recording, and how to join the workstream: blog.farmgpu.com/ocp-scaling…
🌾🚜
Everyone benchmarks GPUs. Almost no one benchmarks the cluster.
@lebanonjon on @SDxCentral: storage, fabric, and the boring stuff that decides whether your H100s actually earn their keep.
sdxcentral.com/video-intervi…
🚜 The OCP just published the first real framework for DC-powered AI data centers. Most of the industry hasn't read it yet.
On March 30, the OCP DCF Power Distribution sub-project — co-led by ABB and Eaton, with NVIDIA, Google, Microsoft, Amazon, AMD, Siemens, Schneider, Vertiv, and 180+ others on the workstream — shipped a white paper on Low Voltage DC power distribution for AI factories.
This isn't a research curiosity. It's the architectural foundation for every NVIDIA 800V DC rack Jensen has been talking about, and the reason Mt. Diablo exists.
The thesis in one sentence: AC has a 100-year head start, but it can't carry a 1+ MW rack without losing 5–10% to conversion stages and choking on 30→100% load swings in milliseconds. So the industry is rediscovering DC.
The numbers worth stopping for:
→ ORv3 HPR power shelves went from 33 kW (v1, AC) → 72 kW (v2, AC) → 100 kW (v4, DC) in the same 1 OU footprint. A 39% density jump, almost entirely from deleting the AC/DC conversion stage.
→ Moving the BBU and CBU out of the IT rack into the Mt. Diablo sidecar reclaims 12–16 OU of compute volume. Another 27–36% of usable rack space, given back to GPUs.
→ AI training availability target: 99.9%. Inference: 99.999%. The catch nobody talks about: DC distribution cannot use UPS maintenance bypass. Redundancy has to live at the power block level. This rewrites how you think about N+1.
→ AI workloads ramp 30→100% in milliseconds. The CBU (capacitor backup) responds in sub-ms. AC architectures can't keep up. Grid operators have noticed.
If your data center can't physically fit a Mt. Diablo / 800V DC rack, you can't run the next NVIDIA generation at full density. The 415V/480V room is already a stranded asset for hyperscale AI.
Read the whitepaper: opencompute.org/documents/dc…
🌾🚜
B200 bare metal performance. Spins up in minutes.
Good news
FarmGPU + @RunPod Instant Clusters: B200 HGX, 800G open Ethernet fabric, 8x PCIe 5.0. NVMe per node via @Solidigm
No procurement queue. No hyperscaler tax. Just the hardware.
Bad news
we are sold out 😭 need more GPUs
blog.farmgpu.com/b200-cluste…
FarmGPU retweeted
NVIDIA's GPU fault remediation tool went production-ready this week. It's called NVSentinel, it's open-source, and I think it's worth understanding how it works because it draws a clean line between two very different layers of the infrastructure stack.
NVSentinel sits inside Kubernetes. It reads GPU health signals from the NVIDIA driver via DCGM, catches things like ECC errors and driver crashes, and then takes automated action at the scheduler level: cordon the node, drain workloads, hand off to a repair workflow. For teams running NVIDIA GPUs on Kubernetes, that automation is real and useful.
What caught my attention is where it stops. NVSentinel operates above the OS. It can tell you a GPU is failing and pull it out of rotation, but it doesn't go below the driver layer to diagnose root cause, it doesn't talk to BMCs, and it doesn't correlate failures across a fleet. The repair workflow it triggers is a handoff to an external system that has to actually figure out what happened and fix it. That lower layer, actual hardware-level diagnostics and recovery, is what we're focusing on at Cosmic.
I like that NVIDIA is raising the baseline here. It makes the conversation about where the real complexity lives a lot easier to have.
What does your GPU fault response look like today? Are you handling it at the Kubernetes layer, the hardware layer, or both?
AI broke the old cloud model.
General-purpose cloud was built for cheap compute abstraction. AI infra is different: storage, fabric, and power density now drive the economics.
If your GPUs wait on I/O, your cloud model is wrong.
blog.farmgpu.com/the-great-i…
We’re testing NIXL, Dynamo and new kvcache bench now. Very different world where GPU is the star of the show and they become data monsters instead of just compute monsters
NVIDIA STX is more than just a new storage device. It represents a redesign of how AI systems move, access, and manage data. Traditional storage architectures were built for reliable, large-scale data storage, but agentic AI and long-context inference require different capabilities. These systems need to retrieve data quickly, maintain context across multiple steps, and access information continuously during inference workflows. Under these conditions, conventional storage can become a bottleneck: increased latency, slow data transfer, and decreased GPU efficiency. STX aims to bridge this gap.
Essentially, STX functions as a high-speed data layer positioned between GPUs and standard storage infrastructure. Its purpose is to bring data closer to computing resources, accelerate read/write operations, and reduce data movement overhead. This allows GPUs to spend less time waiting for data, enabling AI models to handle long contexts, multi-step reasoning, and real-time tasks more efficiently.
STX is not just about improving storage performance by optimizing the efficiency of the entire AI infrastructure. Future AI systems will be defined not only by raw compute power but also by how quickly data can be delivered, how well context can be maintained, and how effectively the inference pipeline is optimized.
