No curl, wget or nc on a locked-down host? Try this next time - (exec 3<>/dev/tcp/HOST/443) && echo OPEN || echo CLOSED - On bash 5.2.21: dash (Ubuntu's sh) fails, - bash --posix still works, and HTTP/1.1 hangs without Connection: close. Read more here from our blog with @sphirani ! one2n.io/blog/test-tcp-conne…

1/5 SGLang v0.5.21 shipped last week with updates across prefill performance, disaggregated serving, prefix caching, AMD GPU support and decision models. A few changes stand out if you're building or operating LLM inference infrastructure. github.com/sgl-project/sglan…

Understanding memory on NVIDIA’s GH200 and GB200 superchips! > An AI model runs on a GPU. The GPU keeps the model in its own memory, called HBM. HBM is very fast, but it is small. > NVIDIA’s GH200 also lets the GPU use the CPU’s memory. CPU memory is much larger, but slower. Together they give up to 624 GB. Can an AI model use all 624 GB at the same speed? No. This thread explains why, in simple steps.

Problem-2 OOMKills at traffic peaks: memory_limiter first, with GOMEMLIMIT aligned to the memory budget. Problem-3 Health-check noise: filter routes using http.route. Problem-4 Missing pod metadata: use k8sattributes, but avoid enriching every span with high-cardinality fields you don't need.

Blog by Sanket (Our in house OTEL hero) tells us Six OTel Collector problems on Kubernetes you commonly face, and the processor setting fixes that might sit behind them. Problem-1 Backend 429s: use batch to reduce export request overhead. Put it after processors that filter or drop telemetry. Example: 30s timeout, batch size 1000.

If models releases are bothering you, here's something - 1/3 Sonnet 5.5 (28 Sep), GPT-6.1 Sol (29 Sep) and Gemini 4 Argon (30 Sep, gated access) launched around the same price: $2/M input tokens and $10/M output. Same list price doesn't mean the same cost per task. Interested? let's dive in

AI Observability Metrics! Time to first token and tokens per second are two different bottlenecks Time to first token and tokens per second hit different hardware limits. We're writing a small 5-part series on inference engineering from the silicon up. Part 1: the two phases of every LLM request - prefil and decode

A Postgres publication setting silently dropped CDC events for weeks! 1/4 Four tables in a Postgres to Debezium CDC pipeline stopped updating in prod. Flush lag 0.02 s, Debezium 62 ms behind source. Every health signal looked normal, and nothing alerted for two weeks.

Things that rolled last week that you should check out this week! Cloudflare Traces is in open beta ( since 2 Oct): automatic OpenTelemetry spans across WAF, cache, routing, Workers and origin, W3C traceparent passed through, OTLP export to any backend. Billing moves to $0.25/GB ingested on 1 Dec. blog.cloudflare.com/cloudfla…

We are a full house today for our AI Observability Workshop! Saturday morning kicks off by learning how to instrument applications across RAG to Agentic Workflows , the gotchas and so much more!

@sphirani - our workshop coordinator here with the gang 🙌

Inference Engineering Decisions for Leaders by One2N - 4/5 What limits concurrent serving of users? A check for this week: vLLM's startup log prints the KV cache size in tokens and the maximum concurrency at your max-model-len. Compare it with your peak concurrent requests before sizing up the GPU , what do you find? On most LLM servers it's KV cache memory. Every in-flight request stores a key and a value vector per token, per layer, per KV head. For Llama 3.1 8B in BF16 that comes to 128 KiB per token, so one 8,000-token conversation holds about 1 GB of GPU memory. Older servers reserved the maximum length for every request up front. A request holding 1,500 tokens in an 8,192-token reservation leaves 82% of it empty, and the PagedAttention paper (SOSP 2023) measured 60–80% of KV cache wasted in those systems. Paged allocation hands out 16-token blocks as the sequence grows, which brings waste under 4%, and to about 0.5% in that example. Most open engines now work this way; custom serving code often doesn't. Tomorrow: turning these numbers into a capacity plan. #LLMInference #GPU #AIInfrastructure

Inference Engineering Decisions for Leaders by One2N - 3/5: Question: Do you use model serving frameworks (HuggingFace, vLLM, SGLang) for your production traffic on agentic applications, and whether it has been load tested with real prompt lengths or only checked for correct answers? Hugging Face Transformers, or a FastAPI wrapper around PyTorch, gives correct answers and poor throughput. These libraries run the model but do nothing to schedule concurrent requests or manage KV cache memory. Serving engines such as vLLM, SGLang and TensorRT-LLM add three things: - KV cache allocated in small blocks as each sequence grows, - continuous batching that adds and removes requests at every decode step, and - inference kernels with FP8 and 4-bit support. In the vLLM team's 2023 benchmarks on LLaMA-7B and 13B, that combination delivered 14 to 24 times the throughput of Transformers on the same GPU. Every project has moved on since, so read it as the size of the gap in 2023. TGI, Hugging Face's own server, is now in maintenance mode, and its docs point to vLLM or SGLang. Tomorrow: KV cache memory, where most of that gap comes from. #LLMInference #vLLM #MLOps

Part 2 of 5 on Inference Engineering Decisions for Engineering Leaders: Theme - where do I run the model? Question - Are you able to group last month's model calls by tasks performed? The share that doesn't need a frontier model is where self-hosting would start , and decide what tasks could use a lower model? A per-token API costs the same per request at any volume. A self-hosted GPU costs the same per hour at any load. The comparison comes down to how many tokens your GPUs serve per hour averaged across nights and weekends. Sized from peak traffic, self-hosting looks cheaper than it will turn out to be. Three things move the break-even towards self-hosting. - The first is a large share of simple requests, such as classification, extraction and routing, that a small open model on one GPU can handle. - The second is fine-tunes, which providers charge more to serve: in early 2025 OpenAI listed fine-tuned GPT-4o at $3.75 per million input tokens against $2.50 for the base model. - The third is data that must stay with your own cloud provider and region. The usual result is a split, with the frontier API kept for hard requests and a small self-hosted model taking the high-volume simple ones. #LLMInference #CloudCosts #AIInfrastructure

Inference Engineering Decisions for Leaders by One2N - 1/5: A check for this week that you can run at your org: Are you able to, for one AI feature, find the cost of a single request and its p95 latency at twice today's traffic? This week, we'll be writing five short posts on running LLMs in production, one decision per day, in the order a team usually meets them. Part 1 is - diagnosis. When an LLM feature degrades under real traffic, three symptoms tend to appear. Latency rises - p95 latency holds, then rises sharply once requests start queueing behind the current batch. Throughput flattens - while the GPU still has compute to spare, because a server that allocates KV cache memory badly can only fit a few requests at once, and each decode step re-reads the full weights for those few. Cost scales linearly with users - because the deployment scales by adding replicas instead of fitting more requests onto each GPU. All three usually come from the serving layer. Before adding GPUs, run a stepped load test to two or three times today's peak, and read GPU load from DCGM's SM activity. nvidia-smi's utilisation figure only reports whether any kernel was running, so it overstates how busy the GPU is. Tomorrow: whether you want to keep paying per token to model, providers or change your mind? #LLMInference #AIInfrastructure #PlatformEngineering