One platform, every GPU cloud. Deploy H100s to B300s across multiple providers. On-demand, no lock-in, from a single dashboard.
Remote
Joined January 2023
- Tweets6.5K
- Following69
- Followers118K
- Likes4.7K
A fine-tuning run that flew Monday crawls by Wednesday. No crash, no warning. The GPU just went from 4,800 GB/s to 50 GB/s.
Dedicated VRAM runs 2,000 to 4,800 GB/s. Shared system RAM tops out at 50 to 100 GB/s. Spill into RAM and you eat a 20 to 100x hit, with nothing in the logs to flag it.
By hardware:
- H100 HBM3: 3,350 GB/s
- H200 HBM3e: 4,800 GB/s
- B200 HBM3e: 8,000 GB/s
- DDR5 quad-channel RAM: about 90 GB/s
A 70B model in FP16 needs about 140 GB for weights alone. Go over VRAM and it shows: 10 tokens/sec on an RTX 4090, 28 on an M4 Max with unified memory, same model. Training jobs stretch 2 to 5x, silently.
What model size hit you first?
R2's reasoning trace can hit 40,000 tokens before it answers, against about 400 for a normal reply. That's 100x more pressure on the KV cache, the GPU memory holding the conversation. Requests queue even when GPU utilization looks fine: the bottleneck is memory, not compute.
Fix: MLA attention instead of GQA. On a 61-layer model over a 30,000-token trace: 2.1 GB under MLA vs 15+ GB under GQA.
Precision picks your hardware:
- FP16: 8x H200, 1,128 GB
- FP8: 8x H100, 640 GB, under 1 point lost on MATH-500, under 2 on AIME
- INT4: 6-12% lost on MATH-500, up to 18% on AIME. Skip it.
vLLM v0.9+: --enable-expert-parallel, --enable-chunked-prefill, and set --max-model-len to your 95th-percentile trace, not the theoretical max.
A 70-billion-parameter model needs about 140 GB of memory just to load. Rent a GPU with half that and the job crashes mid-training, after you've already paid for it.
Most teams rent first and check if the model fits later. Backwards.
Spheron's free GPU Recommender fixes that: type in any Hugging Face model name and get the VRAM you need and which GPU tier actually fits, before you rent.
It helps you:
- Compare GPU tiers using real memory math instead of guesswork
- Catch a card that's too small before you pay for it
- See which smaller model version fits if the full one doesn't
H100 is "1.64x faster than A100" on paper. Your last Llama 3 70B fine-tune probably got about half that.
- BF16, no FP8: 2.1x faster. Matches the paper.
- FP8 switched on: 3.4x.
- NVLink 4.0 moves 900 GB/s vs A100's 600 GB/s. Shows up once you scale past one GPU.
Catch: H100 PCIe ships HBM2e at 2,000 GB/s, not the 3,350 GB/s SXM number everyone quotes. That cuts its edge over A100 PCIe to 1.03x.
W4A8 on H100 needs vLLM 0.4+, TensorRT-LLM 0.10+, or PyTorch 2.1+ with transformer_engine.
Know which variant you're renting and which mode you're actually running.
L40 draws 300W. L40S draws 350W. A 50W difference that looks small on a spec sheet and isn't small on a rack power budget at scale.
For that extra 50W you get a Transformer Engine, automatic FP8 recasting, and roughly double the dense FP16 throughput, 362 TFLOPS against L40's 181. Both cards carry the identical 48GB of GDDR6 ECC at 864 GB/s, so this is a pure compute decision, not a memory one.
If you're power-constrained and not running FP8 workloads, the plain L40 is the more honest choice. Otherwise the 50W is cheap.
H100's headline bandwidth advantage over A100 is 1.64x, 3,350 GB/s against 2,039 GB/s. That number only holds for the SXM5 version.
The PCIe variant of H100 ships HBM2e at 2,000 GB/s instead of HBM3, which narrows its inference bandwidth advantage over A100 PCIe to about 1.03x, functionally no advantage at all.
Check which H100 you're actually quoted before you buy the bandwidth story. SXM5 and PCIe are not the same chip wearing different connectors.
Caching KV data sounds free until you look at where it has to live. GPU HBM costs about $30 per GB per month. CPU DRAM runs about $0.50. NVMe SSD drops to $0.10. Redis-backed storage can go as low as $0.05.
That roughly 600x spread between HBM and Redis is the economic case for a cache-sharing layer like LMCache: move reusable context off the most expensive tier onto something orders of magnitude cheaper.
Multi-node setups push cache hit rate from about 60% on a single vLLM node to about 95% shared across nodes.
On Llama 3.1 70B, about 80% of wall-clock time sits in the decode phase, which is bandwidth-bound, not compute-bound. That's the opening for running prefill and decode on different GPUs instead of the same expensive card for both.
L40S runs prefill 2.7x slower than H100, but it also costs $0.72/hr against H100's $2.57/hr. If your traffic is decode-heavy, a slower prefill card matters a lot less than what you're paying for the GPU doing most of the work.
Worth testing before you assume uniform H100 everywhere is the safe default. It's the expensive one.
Single-node vLLM prefix caching gets you about 60% cache hit rate. Add multi-node LMCache and it climbs to 95%, with a 15x throughput gain at 200 concurrent requests and 80% system-prompt reuse.
On a 128K-context conversation, TTFT drops from 11 seconds to 1.5 seconds by the 10th turn, 7.5x faster. Prefill compute cost for 1,000 requests at an 80% hit rate falls 69%, from $17.47 to $5.40.
The storage math behind it: GPU HBM runs about $30/GB-month, CPU DRAM $0.50, NVMe SSD $0.10, Redis $0.05. Cache what's reused, don't just hoard it in HBM.
$3.31/hr: Spheron's H200 spot price as of October 2026 (subject to availability, varies by region).
Spot costs less than a reservation, but capacity can get reclaimed and interrupt your job. Before you approve budget to move to H200, check your own breakeven first.
Find it:
- Pull monthly token volume from usage logs
- Multiply by months left on your plan
- Estimate GPU-hours for that volume, multiply by $3.31
- Compare totals: under breakeven, hold off
A newer chip doesn't change that math. Bursty or latency-sensitive workloads may fit reserved or on-demand pricing better than spot.
Run your own numbers free
AWS Trainium 3 claims $0.28 per million tokens vs H200 on-demand's $0.84 and H200 spot's $0.61. A real saving over spot alone, on paper.
The catch is breakeven: about 109 billion tokens to recoup the switch against H200 spot. At 1 billion tokens a day that's 109 days. At 10 million a day it's 30 years. Migration itself runs about $36,000, three weeks and two engineers at $150/hr, more for custom attention kernels.
Trainium 3 carries 144GB HBM3e at 4.9 TB/s per chip vs H200's 141GB at 4.8 TB/s. Run your own daily volume through that breakeven first.
