Pinned Tweet
We often get asked about why we are called One2N well, here’s @chinmay185 & @jdk2588 sharing the story.
Our mission: help you go from 1 to N, wherever you need scale. Watch the video to hear it straight from the founders!
What do you do when APM cannot trace the TCP library that more than 200 of your services share?
A real-money gaming operator had this problem. The platform handles more than 180M transactions a day, with peaks of 600,000 requests per second on more than 30,000 VMs. It took 10 to 15 minutes to detect business problems.
The shared library was built on Netty, with a binary protocol and asynchronous event loops. Commercial APM agents could not follow requests through it. Nobody had changed it in five years, because changes were risky.
Our team made the following changes:
• They added OpenTelemetry client and server spans inside the shared library.
• They carried W3C trace context through the binary protocol. One framework change gave end-to-end traces for every service.
• Used AI to map the Netty pipeline and the event-loop hand-offs before the code change. Then they benchmarked the change on Java 8.
• They replaced poll-based checks with four streaming metrics: spins, bets, wins and revenue.
Detection now takes less than 60 seconds, and traces cover 88% of critical user journeys. The old stack stayed live during validation, so the team could roll back each step.
Read our whole case study here - one2n.io/case-studies/modern…
Understanding FP8 and NVFP4 for AI Inference , and why does it matter ?
What do FP8 and NVFP4 actually mean, and why do they matter for AI inference?
FP stands for floating point, a way to represent numbers. The number after FP tells you how many bits are used to represent each value.
FP8 uses 8 bits. FP4 uses 4 bits.
NVFP4 is NVIDIA's 4-bit floating-point format, designed for efficient AI computation on Blackwell GPUs.
Let's see what this means for memory and inference.
Why does using fewer bits matter?
First, memory capacity. Smaller weights leave more GPU memory available for the KV cache, which stores information about the tokens processed in each conversation. That can allow more concurrent requests.
Second, data movement. Reading fewer bytes of weights can improve inference throughput, depending on the workload and hardware.
But the GPU must support the format efficiently. NVIDIA Blackwell GPUs support native FP4 Tensor Core operations. Hopper GPUs, including H100 and H200, support FP8 but lack native FP4 Tensor Core support.
There's a trade-off: reducing numerical precision can affect model quality. The impact depends on the model, quantisation method and task.
For inference capacity planning, follow this order:
Test the smallest format that meets your quality requirements.
Calculate memory for weights, KV cache and runtime overhead.
Choose the GPU that meets your throughput and latency targets.
The goal is to fit more useful inference into the available hardware without compromising the output your workload needs.
Want our whitepaper with the calculations? Send us a message.
#LLMInference #AIInfrastructure
Understanding total and active parameters
Four open-weight models were announced in the first week of October. Each model card shows two numbers. Together they decide the hardware that you need to run the model yourself.
The two large models arrive later this month: Mistral Large 4 (1.05T total, 52B active) and Reflection's Beam (501B total, 23B active).
For these, the number format decides the GPU:
• FP8, 1 byte for each parameter: Mistral Large 4 needs about 1.05 TB. Eight H200s (1,128 GB) leave only about 78 GB for users.
• NVFP4, about half a byte: about 590 GB. Eight B200s (1,440 GB) leave about 850 GB.
Understanding what a desktop AI machine can serve
NVIDIA's DGX Spark has 128 GB of memory, and NVIDIA says it runs models up to 200B parameters.
Quiz: how many tokens per second does a 70B model make on it? The answer is in this thread, part 4 of our inference series.
For each token, the model reads all of its weights. So the speed for one user is the bandwidth divided by the size of the weights:
• 70B model in NVFP4, about 40 GB: about 7 tokens/s. That is the quiz answer.
• Llama 3.1 8B in FP8, about 8 GB: about 34 tokens/s
• Kolibri-1, a mixture-of-experts (MoE) model: about 79 tokens/s
An MoE model uses only a few of its parameters for each token. Kolibri-1 has 78B, but only about 3.5B are active, so it reads less memory.
So the model fits, but it answers slowly. DGX Spark is good for development, tests and batch jobs on private data. It is not for many users at the same time.
Before you buy any machine for AI, divide its bandwidth by the size of your model.
Which model would you run on 128 GB? Reply. Part 5 tomorrow: FP8 and NVFP4.
Inference puzzle:
Llama 3.1 70B, FP8 weights (~70.6 GB), one H200 with 4.8 TB/s HBM bandwidth. Ignore KV-cache reads and kernel overhead.
What's the bandwidth-limited maximum tokens/s for one stream? What changes at batch size 32?
We'll post the calculations tomorrow, drop your answers meanwhile!
If you're routing work across providers, run the same labelled task set through each model. Record:
- Input and output tokens
- Cache hits
- Pass rate against your acceptance criteria
- Cost per successful task
- Latency at the percentile that matters to your users
Then route based on the cost of getting a correct result, not just the price per million tokens.
A gateway that records per-request usage and outcomes makes this analysis much easier to maintain.
#LLMOps #FinOps #GenerativeAI