@usemuna

Structurally cheaper inference for open models.

muna.ai
Joined December 2021
  • Tweets11
  • Following3
  • Followers139
6Media
Most inference GPUs are ~70% idle. It's not a kernel problem, or a traffic problem. It's a placement problem. To prove it, we built an inference service that co-locates models on a GPU. 5 co-located models. 14 days. 2.5B tokens served. $17k/mo saved.

The Next 3× in Inference Won't Come From Faster Kernels

If you work in AI infrastructure, you live with two facts that seem to contradict each other: there is a massive GPU shortage, yet the GPUs we have sit at roughly 30% utilization.

Article · @usemuna
4
1
1
13
1,382
Load more