https://xitter.cf/t.co/BXAeZpbDtP builds software that optimizes GPU clusters for fault tolerance, deterministic performance and increased utilization.
Palo Alto, California
Joined April 2021
- Tweets392
- Following112
- Followers98
- Likes244
Pinned Tweet
At scale, failures are routine. Losing the work has been routine too. Not anymore.
LinkedIn runs Clockwork.io in production. Together.ai is bringing TorchPass to its GPU cluster customers. SemiAnalysis measured the payback: goodput loss cut from 14% to under 3%.
Today we're launching an industry first, in preview: TorchPass Snapshots in addition to Asynchronous Application checkpoints. And announcing $31M in new funding.
๐น TorchPass Snapshots: save a whole running training job, across every node, with no code changes. Free the GPUs now. Resume later. Nothing lost. Platform teams stop waiting for application owners to add checkpointing.
๐น Async Application Checkpoints: checkpoint while you train, not instead of training. Save more often, lose less when something breaks, and get fresh weights to Reinforcement Learning rollouts sooner.
Our AI fault tolerance is already proven in production:
โ
LinkedIn runs Clockwork.io LinkPass across its AI infrastructure fleet.
"In aggregate, Clockwork.io prevents tens of thousands of GPU-hours of downtime per month across our fleet." Raghu Hiremagalur, SVP, CTO Infrastructure, LinkedIn.
โ
@togethercompute is bringing TorchPass to market as a service on its GPU Clusters.
"Our customers grade us on goodput, the share of their GPU-hours that actually move the model forward." Pavneet Singh Ahluwalia Ahluwalia, Product Lead, Together.ai
โ
WhiteFiber is expanding Clockwork.io across its global GPU footprint. "We bring clusters up faster, and a customer's first training run lands on a fabric validated end-to-end, not just powered on." Tom Sanfilippo, CTO, WhiteFiber.
Quantifiable payback: "In our ClusterMAX, TorchPass cuts training goodput loss from 14% to under 3% for a gold-rated neocloud." Dylan Patel, @SemiAnalysis_ , @dylan522p. More than 11 points of GPU capacity back, measured independently.
New funding: $31M with @PremjiInvest, @Wing_VC, Seligman Ventures, New Enterprise Associates (@NEA), @etisalatandae
Pressure tested at scale with some of the worldโs largest enterprises, neoclouds and hyperscalers. The job never stops.
Stop paying for job restarts. Preview access opens today, and we'll kill a GPU and a link live at PyTorch Conference with Together.ai. Schedule a demo and preview access in the first comment ๐
#AIInfrastructure #FaultTolerance #DistributedTraining #GPU #PyTorch
๐๐ง๐ฒ๐จ๐ง๐ ๐ซ๐ฎ๐ง๐ง๐ข๐ง๐ ๐ฅ๐๐ซ๐ ๐ ๐ญ๐ซ๐๐ข๐ง๐ข๐ง๐ ๐ฃ๐จ๐๐ฌ ๐ค๐ง๐จ๐ฐ๐ฌ ๐ญ๐ก๐ ๐๐๐ฆ ๐ฉ๐๐ ๐: ๐จ๐ง๐ ๐ฅ๐ข๐ง๐ค ๐๐ฅ๐๐ฉ, ๐๐๐ ๐๐๐ฎ๐ฅ๐ญ, ๐จ๐ซ ๐ง๐จ๐๐ ๐๐๐ข๐ฅ๐ฎ๐ซ๐, ๐๐ง๐ ๐ญ๐ก๐ ๐๐ง๐ญ๐ข๐ซ๐ ๐ฃ๐จ๐ ๐ข๐ฌ ๐๐จ๐ฐ๐ง.
One dead rank stalls every collective until the NCCL watchdog fires. Then checkpoint and restart: 45-90 minutes across every GPU in the job.
Next week at PyTorch Conference North America, Together AIโs Pavneet Ahluwalia and Clockworkโs Gavin Cohen walk through a training failover system built to keep the job running through all of it, and how to tell when live migration beats checkpoint-restart.
๐ฃ๏ธ ๐๐จ๐ ๐๐ญ๐ก๐๐ซ ๐๐ + ๐๐ฅ๐จ๐๐ค๐ฐ๐จ๐ซ๐ค ๐ฃ๐จ๐ข๐ง๐ญ ๐ฌ๐๐ฌ๐ฌ๐ข๐จ๐ง
Self-Healing PyTorch Training Through Live GPU Migration
Wed Oct 21 ยท 12:20 PM PDT ยท Room LL20CD
โฐ ๐๐ข๐ ๐ก๐ญ ๐๐๐ญ๐๐ซ, ๐ฐ๐๐ญ๐๐ก ๐ข๐ญ ๐ก๐๐ฉ๐ฉ๐๐ง: ๐๐๐ฏ๐๐ซ ๐๐ข๐ญ ๐๐๐ซ๐จ
Lin She, Clockwork.io
Wed Oct 21 ยท 12:45 PM PDT ยท Demo Theater, Community Expo
๐๐๐ ๐๐ก ๐๐ ๐๐๐ก๐๐๐ ๐๐ ๐ค๐ ๐๐ข๐๐ ๐ ๐๐๐ก๐ค๐๐๐ ๐๐๐๐๐ ๐๐๐ ๐ ๐๐๐ค๐๐ ๐๐๐๐๐ ๐๐๐๐ ๐ ๐๐ข๐๐๐๐๐ ๐๐๐, ๐๐ ๐๐๐๐๐๐.
๐ ๐๐จ๐จ๐ญ๐ก ๐๐, ๐๐จ๐ฆ๐ฆ๐ฎ๐ง๐ข๐ญ๐ฒ ๐๐ฑ๐ฉ๐จ. Bring your failure modes. Book a 1:1 or a private live TorchPass demo.
#PyTorchConference #DistributedTraining #FaultTolerance #GPUClusters #NCCL
โก๏ธ Book time: na2.hubs.ly/H08qh5Z0
Read the press release on X: na2.hubs.ly/H08byd60
Schedule the unstoppable demo: clockwork.io/contact/
Last night we kicked off @CoreWeave 's Fully Connected 2026 โ a reception with robots pouring drinks, games, and some of the sharpest full-stack AI infra engineers all in one room.
๐๐๐ฒ ๐ ๐ค๐ข๐๐ค๐ฌ ๐จ๐๐ ๐ญ๐จ๐๐๐ฒ ๐ฐ๐ข๐ญ๐ก ๐ข๐ง๐ฌ๐ฉ๐ข๐ซ๐๐ ๐ข๐ง๐ง๐จ๐ฏ๐๐ญ๐ข๐จ๐ง๐ฌ ๐ข๐ง ๐ฉ๐ซ๐จ๐๐ฎ๐๐ญ๐ข๐จ๐ง ๐๐๐ซ๐จ๐ฌ๐ฌ ๐๐ฏ๐๐ซ๐ฒ ๐ข๐ง๐๐ฎ๐ฌ๐ญ๐ซ๐ฒ, and culminates with ๐ ๐ซ๐๐๐ฅ ๐๐๐ญ๐ญ๐ฅ๐ ๐จ๐ ๐ญ๐ก๐ ๐๐จ๐ญ๐ฌ.
Catch us at the Clockwork booth in Pioneer Pavilion and see how we are built for production scale reliability, keeping AI runs alive through infra failures.
#FullyConnected #AIInfra #Battleofthebots #CoreweaveEvent
๐๐ ๐ค๐๐๐ฉ๐ฌ ๐ฆ๐จ๐ฏ๐ข๐ง๐ โ ๐๐ฏ๐๐ง ๐ฐ๐ก๐๐ง ๐ ๐๐๐ ๐๐๐ข๐ฅ๐ฌ ๐จ๐ซ ๐ ๐ฅ๐ข๐ง๐ค ๐๐ซ๐จ๐ฉ๐ฌ.
At @CoreWeave's Fully Connected (Sept 29โOct 1, San Francisco), we're showing what Workload-Aware Fault Tolerance actually looks like: FleetLens audits the fleet and pinpoints the fault. TorchPass and LinkPass keep distributed workloads running.
๐๐ ๐๐๐ข๐๐๐-๐๐๐๐ ๐ฆ๐๐ข ๐ก๐ ๐๐๐๐ ๐ ๐๐ ๐๐ก ๐๐ ๐๐๐ก๐๐๐.
๐ Book time with the team: na2.hubs.ly/H080KZ30
#FullyConnected #AIInfrastructure #GPU #TorchPass #YOCO #LinkPass #workloadaware #faulttolerance #DARE
๐๐ ๐๐จ๐ฎ๐๐ฅ๐-๐๐๐ซ๐ ๐ฒ๐จ๐ฎ: ๐ฎ๐ง๐ฐ๐ข๐ง๐ ๐๐ง๐ ๐จ๐ฎ๐ญ-๐๐จ๐ฆ๐ฉ๐ฎ๐ญ๐ ๐๐ญ ๐ญ๐ก๐ ๐ฌ๐๐ฆ๐ ๐ญ๐ข๐ฆ๐.ย ๐ท ๐ง
Join the Clockwork team at the Wine & Cheese Bar โ good pours, good bites, and a live look at GPU fault tolerance that doesn't cost you speed. Bring your worst failure story; we'll show you the 1.8-hour recovery tax nobody budgets for โ and how to erase it.
๐
Tuesday, Sept 29 ยท 5:00โ7:00 PM ยท Pioneer Pavilion
๐๐๐๐ to have fun and learn. See you there.
@CoreWeave
#FullyConnected #AIInfrastructure #DARE
๐๐๐ญ๐๐๐ญ๐ข๐จ๐ง, ๐ซ๐๐ฆ๐๐๐ข๐๐ญ๐ข๐จ๐ง, ๐๐ก๐๐๐ค๐ฉ๐จ๐ข๐ง๐ญ๐ข๐ง๐ โ ๐ง๐๐๐๐ฌ๐ฌ๐๐ซ๐ฒ, ๐๐ฎ๐ญ ๐ง๐จ๐ญ ๐๐ง๐จ๐ฎ๐ ๐ก ๐๐จ๐ซ ๐ฌ๐ญ๐ซ๐จ๐ง๐ ๐ ๐จ๐จ๐๐ฉ๐ฎ๐ญ.
At AI Infra Summit, Clark Zinzow of @togethercompute laid out the four goodput pain points they see most:
โข ๐ถโ๐๐๐๐๐๐๐๐ก ๐๐๐ ๐ก๐๐๐ + ๐๐๐๐๐๐๐ข๐ก๐ โ even on high-performance scale-out storage systems, restoring can take minutes, while recomputing progress since the last checkpoint adds tens more.
โข ๐ป๐๐ก-๐ ๐๐๐๐ ๐ก๐๐๐๐๐๐๐๐ โ always-on spares are expensive to hold idle; cold spares are too slow to pull into the cluster.
โข ๐ถโ๐๐๐๐๐๐๐๐ก ๐๐๐๐๐ข๐๐๐๐ฆ โ checkpoint more often to lose less, and the overhead eats into your goodput.
โข ๐บ๐๐๐ฆ ๐๐๐๐๐ข๐๐๐ โ every health check passes, yet the job quietly slows or stalls.
How we close all four is in Part 4, linked in the comments.
#AIInfraSummit
We dare you: pull a link. Kill a GPU. Then watch the job keep running.
At @CoreWeave's Fully Connected AI Cloud conference, stop by the Clockwork booth and see how AI jobs survive planned and unplanned failures. LinkPass reroutes around the dead link, TorchPass live-migrates the work to a healthy GPU โ no rollback, no recompute, no lost progress. SemiAnalysis clocked it: 405 minutes vs. 818 minutes for checkpoint-restart. 2ร faster, zero lost steps.
๐Fully Connected 2026 ยท Sept 29โOct 1 ยท San Francisco ยท Clockwork booth
Take the dare. We'll be ready.
#FullyConnected #AIInfrastructure #GPU
Checkpoint-and-restart isn't fault tolerance. It's a tax on goodput โ pay to restore, then pay again to recompute.
At #AIInfraSummit with @togethercompute : move a live job off failing GPUs, no rollback. Benchmarked by SemiAnalysis, TorchPass recovers in 405 min vs 818 min (checkpoint restart).
Next clip in the thread.
Fault tolerance isn't a niche concern for a handful of frontier labs anymore. It's the conversation every team running AI workloads at scale is having.
๐๐๐ซ๐'๐ฌ ๐ญ๐ก๐ ๐ง๐ฎ๐๐ง๐๐ ๐ฆ๐จ๐ฌ๐ญ ๐ญ๐๐๐ฆ๐ฌ ๐๐ซ๐ ๐ฌ๐ญ๐ข๐ฅ๐ฅ ๐ฆ๐ข๐ฌ๐ฌ๐ข๐ง๐ : detection and checkpointing alone don't ensure strong goodput.
At AI Infra Summit, we showed what that looks like in practice: a network link and a GPU both fail mid-run on a synchronized workload, and the job keeps advancing instead of rolling back. No full restart, no lost work.
On the Data & Models stage, Clark Zinzow (@togethercompute) and Prashanth Thinakaran (Clockwork Systems, Inc.) took the same message further in their joint session, ๐๐ซ๐๐ฌ๐๐ซ๐ฏ๐ข๐ง๐ ๐๐ซ๐จ๐ ๐ซ๐๐ฌ๐ฌ ๐๐ง๐ ๐๐จ๐จ๐๐ฉ๐ฎ๐ญ: ๐๐จ๐ซ๐ค๐ฅ๐จ๐๐-๐๐ฐ๐๐ซ๐ ๐
๐๐ฎ๐ฅ๐ญ ๐๐จ๐ฅ๐๐ซ๐๐ง๐๐ ๐๐จ๐ซ ๐๐ข๐ฌ๐ญ๐ซ๐ข๐๐ฎ๐ญ๐๐ ๐๐ซ๐๐ข๐ง๐ข๐ง๐ . Together AI on health checks and remediation, Clockwork on active job recovery through checkpoint-free live migration โ a packed room hearing exactly why.
Reliability has become everyone's problem to solve. Health checks catch it. Live migration keeps it running.
See it for yourself - link in comments.
๐ And congrats to our raffle winners, including Jason Wilder and Sunil Tagare โ thanks for being part of the conversation this week.
#AIInfrastructure #FaultTolerance #GPUClusters #MLOps #Goodput #AIInfraSummit2026
๐Interactive Demo: na2.hubs.ly/H07W06P0
๐๐ก๐๐ญ๐ก๐๐ซ ๐ฒ๐จ๐ฎ ๐ซ๐ฎ๐ง ๐ข๐ง๐๐ซ๐, ๐๐ฎ๐ข๐ฅ๐ ๐จ๐ซ ๐ญ๐ซ๐๐ข๐ง ๐ฆ๐จ๐๐๐ฅ๐ฌ, ๐จ๐ซ ๐ฆ๐๐ง๐๐ ๐ ๐ญ๐ก๐ ๐ฉ๐ฅ๐๐ญ๐๐จ๐ซ๐ฆ โ ๐ฌ๐๐ ๐๐จ๐ซ ๐ฒ๐จ๐ฎ๐ซ๐ฌ๐๐ฅ๐ ๐ฐ๐ก๐๐ญ ๐ก๐๐ฉ๐ฉ๐๐ง๐ฌ ๐ฐ๐ก๐๐ง ๐ฌ๐จ๐ฆ๐๐ญ๐ก๐ข๐ง๐ ๐๐๐ญ๐ฎ๐๐ฅ๐ฅ๐ฒ ๐๐ซ๐๐๐ค๐ฌ.
Bring your hardest failure scenario to Booth #821 โ where GPU downtime never gets the time of day.
Independently verified. Don't take our word for it.
๐ AI Infra Summit 2026 | Sept 15โ17 | Santa Clara
Book a 1:1 demo โ spots are limited.
#AIInfraSummit2026 #pulledlink #AIInfrastructure #AIworkloads #distributedtraining #GPUinfrastructure #GPU