Guys, I have a daily goal of reading 1 article a day. I do this by discussing the article with my agent, and the focus is on my self-improvement in Machine Learning, Agents, Harness, LLMs, and other things in this universe that amuse me. In this process, we create images to make the rationale crystal clear. So, I decided to start posting my studies and analyses here. Because of that, there's a daily post here that looks automated, but it isn't. I have no intention of selling anything. I'm a studious and hardworking guy, a family man passionate about technology and what's behind it (in a technical sense). I am a human, and I'm here to talk to humans. I'm a builder, working on a few applications as side projects, and soon I plan to share them and invite people to test them out. My focus is B2B. If you're interested in discussing these topics, follow me and I'll follow you back (provided you don't post garbage about politics, gossip, hate speech, and other things that don't help humanity evolve).

Researchers at Microsoft M365 Research and IISc analyzed 121M production model calls from four M365 workloads to size the KV cache agents keep between calls. Holding every live Deep Research session at its peak would need an estimated 501.8 TB of GPU memory. Inference uses that state only 5.6% of the time it stays resident. The study comes from Mayank Arya, Agrim Bari and coauthors at @MSFTResearch and @iiscbangalore. - Shape: in Deep Research, the orchestrator issues 89% of model calls without new user input. Sessions that reach 30 calls carry a median context of 68K tokens, and 80% of context tokens hit the prefix cache. - Idle cap: dropping a session's KV after one idle hour caps the ceiling at 140.8 TB, 3.6x lower. Calls that recompute their context rise 0.46 points, to 4.53%. - Tight memory: in the busiest Deep Research hour, with 4.7 TB (10% of peak demand), least-recently-used (LRU) eviction makes 62.9% of calls recompute. Clairvoyant Belady eviction leaves 31.1%. Keeping an int4 copy in the same budget instead leaves 13.6%. - Routing: a Deep Research session touches 4.5 deployments on average. If KV stays local to each deployment, 20.2% of calls recompute even with unlimited memory. Limits: a Llama-2-70B KV layout stands in for the confidential models. The replay charges no quality or transfer cost for int4 copies. Before you buy more GPU memory for agents, check how long sessions sit idle and where the next call lands. arxiv.org/abs/2610.22863

Trying paperclip Surprisingly good. My Hermes is configuring it. Ridiculous. I thought we were 2 years from fully managing a company with agents. I'm assessing the shrink of 200 people to 20 max + agents. Very promising and optimistic

A 397B model that reviews a 4B model's agent steps matched the 397B-only success rate on τ²-Bench Airline (86.0%, 50 tasks) in 41% of the wall time. SpecAgent, from @ZhijiangG, @YinhongLiu2 and coauthors at @HKUSTGuangzhou, @hkust, @Cambridge_Uni and Yinwang, applies the draft-and-verify idea of speculative decoding to whole tool-call trajectories. - Draft. Qwen3.5-4B executes read-only tool calls for real, so each drafted step sees true observations. - Hold. The draft stops at the first state-changing action (a refund). That action waits. - Verify. Qwen3.5-397B-A17B reads the segment in one prefill pass and accepts the longest prefix it judges valid. A valid step counts even when the 397B would choose another. - Repair. When the fix is one tool call, the verdict carries it. Otherwise the 397B takes one repair turn. Token-by-token decoding is bound by memory bandwidth. Prefill reads a whole segment per weight pass. On Airline, the 397B makes 3.1 calls per trajectory instead of 13.8 alone. With GLM-5, DeepSeek or the 397B behind a cloud API, the cloud model writes 36-50% of its usual tokens and API cost falls to 59-81%. Pass@1 drops 9-13 points. Limits: estimated FLOPs rise 30-87%, so the gain needs memory-bound decode (below about 37 sequences per batch on H20). On AutomationBench SpecAgent recovers only 44% of the quality gap. Put the review gate at irreversible actions. arxiv.org/abs/2610.21463 @Alibaba_Qwen @sgl_project @Zai_org @deepseek_ai

Meta now trains production recommendation models on tens of thousands of AMD MI350X GPUs. Across ten models, throughput matches NVIDIA B200 at a geometric mean of 101.2%. Haoyu Zhang, Chunqiang Tang and 56 coauthors at @AIatMeta and @AMD report what it took (MICRO 2026). Beyond hipifying CUDA code, parity took work in four areas: - Atomics. MI350X has 8 chiplets, each with a private L2 cache. Under contention, atomic-heavy kernels can run over 10x slower than on B200. Kernels sum in shared memory with a tree reduction and issue one atomic per block. - GEMM tiles. Origami, an analytical model, picks tiles in microseconds at 93% of exhaustive autotuning. With thousands of GEMM shapes per model, lookup tables do not scale. - Collectives. Tuning took RCCL, AMD's collective library, to within 10-15% of NVIDIA's NCCL. It started 2x slower. - Memory. MI350X has 288 GB of HBM (B200: 180 GB). Bigger batches, no activation checkpointing and embeddings in HBM gave one model 1.26x over the direct port (1.5x with kernel work). An agent workflow encodes these fixes as skills. It succeeds on over 80% of dozens of production Triton kernels. Limits: the 101.2% compares equal GPU counts at matched batch sizes. GEMMs still run 1.3-1.7x slower, partly because TF32 is emulated with three BF16 GEMMs. Per-model ratios span 82%-130%, and Meta caps B200 at 750 W versus 1000 W for MI350X. Plan a second GPU vendor as a co-design project. arxiv.org/abs/2610.22230 @PyTorch @AMDServer @NVIDIAAI

essa pergunta na verdade não faz sentido. Na verdade são duas perguntas na verdade: - Eu preciso do design do Jony Ive para o que? - Quantos Jony Ive existem e que conseguem fazer um trabalho melhor que a IA? @brunofaggion

A remote KV cache tier can be slower than the GPU it feeds. Muhammad Ibrahim (@MultiverseCompu) serves GLM-5.3, an FP8 mixture-of-experts model with multi-head latent attention, on 8xB300 prefill nodes with vLLM and LMCache. A shared Valkey tier left median time to first token flat. The 16k-token p99 rose to 17.3 s (3.96 s before), and the team rolled the deployment back. The cost model has two terms: - Break-even bandwidth B* = s x R. KV bytes per token (48.4 KB) times node prefill rate (52.2k tokens/s) gives 2.53 GB/s. One Valkey instance peaked at 1.71 GB/s. A 16k-token prompt took 0.88 s to load and 0.31 s to recompute. - A fixed cost per hit, T0 ~ 0.33 s. Below about 17k tokens, recompute wins at any bandwidth. Synchronous loads stall all 8 context-parallel ranks. With one instance on a hit-heavy replay, the engine became an overloaded queue: p99 83.32 s with the GPUs 7% busy. The fixes: - Shard LMCache's KV store: chunk c belongs to rank c mod 8. - Give each rank its own Valkey instance. GET bandwidth rises 6.3x to 10.76 GB/s, with no cluster protocol. - Load asynchronously. - Fix three client bugs that froze engines. On the same replay, p99 falls to 0.67 s. On 200k-token prompts, sharding adds 14% prefill throughput, or 34% with an upstream kernel fix. Each condition ran once. The 17k figure is a model prediction. Test a cache tier against the prefill rate of the node behind it. arxiv.org/abs/2610.21500 @lmcache @vllm_project @valkey_io @Zai_org @NVIDIAAI

Multi-model agents break a serving assumption: every model fits in GPU memory. On one 80 GB GPU, a Llama-3.1-8B planner plus Qwen2.5 and Mistral-7B experts (15.2 GB each, FP16) compete with the KV cache of live sessions. JointServe, by Daijin Hu (Beihang University), runs on SGLang and uses the planner's output stream to choose what to load next. How it works: - Weight layer groups and session KV blocks share one budget. JointServe ranks each by expected recovery work avoided per byte. - An intent tap reads the output stream. When the tool-call header names an expert, prefetch starts while the planner still writes arguments. - Each model keeps its first layers resident. A swap then streams only the remaining layers in 0.14 s, short enough for 98% of SWE-bench tool gaps. - Evicted KV goes to host RAM. Reload takes 0.05 s. Re-prefilling the median SWE-bench context (27.5K tokens) takes 2.64 s. Results (one A800): - 24 GB usable: median session time 21.7% below the best whole-model swap baseline. - 8 GB usable: JointServe finishes in 758 s. Whole-model baselines miss the 35,000 s window. - SWE-bench traces: session time -49%, 2 model swaps per run vs 478 for the baseline. - OSWorld: median time to first token 0.1 s vs 31.7 s. The intent tap needs visible tool-call headers. Tests ran over PCIe 4.0 only. Your agent's call graph is a prefetch schedule you already have. arxiv.org/abs/2610.19822 @sgl_project @vllm_project @Alibaba_Qwen @MistralAI @AIatMeta

Most cost work on coding agents compresses tool output (observations) or routes turns to weak models. Across 600 public coding-agent trajectories, turn count explains 84% of the variance in log cost. Mean observation size explains 4%. Researchers at Penn State re-scored NVIDIA's Open-SWE-Traces (OpenHands, SWE-agent; Qwen3.5, MiniMax-M2.5). Their tool, TurnCost, rebuilds each turn's context and simulates the prefix cache. What cache-aware accounting shows: - 98.6% of input tokens are cache hits. Flat input pricing overstates loop cost by about 7.1x under near-ideal caching (4.0x at 90% cache efficiency). - Median observations are 118-286 tokens. The big build log that motivates compression sits near the 99th percentile. Edits, which echo file context back, carry 51.8% of observation tokens. - Swapping a whole turn to a weak model pays only if its input price beats the strong model's cached-read price (about 1/10 of list). Delegating one observation pays roughly when turns left exceed that price ratio. - Deterministic truncation matched weak-model delegation on cost in all 20 cell-threshold combinations. Cost grows with turns at exponent 1.23, which fits a loop where each turn re-sends every earlier one. The authors call the link associational and do not test removing turns at fixed task success. Before you tune a router, price the cache and count the turns. arxiv.org/abs/2610.15093 @penn_state @NVIDIAAI @OpenHandsDev @OfirPress @Alibaba_Qwen @MiniMax_AI

Not sure why Google treat their pro users like that.

ByteDance runs RDMA under its recommendation inference and training. Most of the engineering went into software above the NIC. ByteExpress (@ByteDanceOSS, @FudanUniversity) sits between ByteDance's RPC framework (Archon) and its RDMA NICs. It has run in tens of clusters for over two years (Toutiao, TikTok). The design has four parts: 1. bemalloc extends rpmalloc so network-bound buffers come from pre-registered blocks. Received buffers pass to app threads without a copy. 2. Sockets to the same host share one queue pair. Fragmenting large messages limits how long they delay small ones. 3. Receive credits: a sender posts a message only when the peer has a free receive slot. 4. A software send window caps bytes in flight per queue pair, above the NIC's built-in congestion control (DCQCN). In training, part 4 made the difference. The production job had 124 parameter servers and 346 trainers. On the busiest trainers, RDMA with hardware congestion control alone had worse P99 fetch and push latency than TCP. Priority flow control (PFC) pauses added up to 1 s in every 30 s. With the window, P99 push latency was 268 ms vs 1.03 s on TCP (-74%), with no PFC pauses observed. Inference at peak, matched load: parameter-fetch P99 of 35.5 ms vs 63.1 ms on TCP, and total CPU 51.3% vs 57.5%. Caveat: the baseline is ByteDance's deployed kernel TCP stack, with no tuned-TCP comparison. If RDMA is on your roadmap, budget for the software above the NIC. arxiv.org/abs/2610.12815

In TraceLab's ~4,300 coding-agent sessions, a median call resends ~119K cached tokens to add 875 new ones. Prefix caching skips that prefill. The CPU still tokenizes the full prompt before the cache lookup can start. Researchers at KTH Royal Institute of Technology served long prompts in vLLM with 99.85%+ of each one cached. The stock tokenizer took 56-76% of time to first token (TTFT). Their fix, GraphTok, runs the encode for long inputs on the GPU as one CUDA graph: - Each thread decides if a pre-token starts at its character, from local Unicode classes and prefix scans, so no serial regex cursor is needed. - For short pre-tokens, a warp picks the next BPE merge with a hardware min over (rank, position). The min keeps reference merge order. - GraphTok sizes buffers per power-of-two capacity class. GPU counters track live work, so the host never allocates or syncs mid-call. Token IDs match the reference bit-for-bit on GPT-2, Llama 3, Qwen 2.5, DeepSeek-V3 and Gemma 3. In vLLM, GraphTok's share of TTFT is 1.4-2.5%. One NVIDIA Dynamo frontend (GPUs simulated) sustains 518 req/s of 121k-token traffic within a 2 s P99. The fastest CPU tokenizer, gigatoken, sustains 296 with its cache cleared per request. For a single request on one engine, GraphTok's median TTFT is 3-18% below gigatoken's. The gain is fleet capacity, plus host cores freed for agent tools. Once prefill is cached, profile your frontend. arxiv.org/abs/2610.13325 @vllm_project @NVIDIAAI @Alibaba_Qwen @AIatMeta

Your agent emits a correct shell command. A hop on the way to the shell can still change it, and the model takes the blame. Researchers at Yanshan, Zhejiang, Nanjing and Aalto universities trace each tool call across 5 hops: tool serialization, host wrapper, shell parse, process interface, target parser. A hop that parses the text again can consume quotes, re-expand variables, truncate the command or drop arguments. Production data: 47,828 shell calls from 261 Windows sessions of Claude Code and Codex. - Claude Code's Bash wrapper changed 12.0% of calls that carry code, escapes or long text. - For 80.7% of calls with merged backslash pairs, the wrong action ran with no error. - 306 of 426 parse errors came from a hop. Each reached the agent as an error in its own text. - An LLM judge blamed the model for 97 of 102 failures, while human labels traced 56 to a hop. On Linux, commands passed to bash -c arrived intact in all 4,012 developer calls. On their benchmark, a passed task costs 2.4x the tokens without the fix. Agents reported 665 of 949 failed runs as completed. Their fix, IntAct, sends the call through a channel the faulty hop does not parse (a sourced file, stdin, an argument vector, PowerShell's -EncodedCommand) or refuses it. It recovers 79.2% of benchmark task pairs lost to a changed call. Before you swap models over agent failures, log what each hop received. arxiv.org/abs/2610.04375 @HaoyeTIAN @ZJU_China @AnthropicAI @OpenAI @Alibaba_Qwen @OpenHandsDev

When a user corrects a coding agent mid-task, the server aborts the request and starts a successor. The old request keeps running until vLLM next checks for aborts, and a cold successor rebuilds a prefix that was still valid. In NVIDIA's tests, 18 vLLM aborts still delivered 170 tokens in total from the abandoned plan, because those tokens sat in internal queues. A long-prefill successor waited 3.21 s. RETIRE splits the request's two roles. Requests keep GPU memory and scheduling, and execution versions hold authority: the permission to publish output or install KV state. One revision runs five stages: - ADVANCE revokes the old version's right to publish. - FENCE halts old GPU launches and closes KV writers. - ALIGN certifies the common prefix the successor may reuse. - RESUME starts the successor on those pages. - RECLAIM frees old memory asynchronously. Results: - Qwen3-32B, H100 PCIe, 90% shared prefix: successor TTFT drops a median 17.1% vs abort plus cold restart. - Under HBM churn, native prefix caching loses the prefix. RETIRE recovers up to 6.87x faster after churn (1.19x over the full interruption). - Densest 120 s of a coding-agent trace (GPT-OSS-20B, no shared prefix): 0 obsolete tokens vs 40,499, and 9/9 final versions advance vs 3/9. - With no revision, guards add a median 1.05% decode overhead. If your agents accept mid-task corrections, count tokens delivered after an abort. arxiv.org/abs/2610.01160 @nvidia @NVIDIAAI @vllm_project @Alibaba_Qwen

why don't we use gitlab?

Great initiative and I'll keep updated on it. But it needs a lot of adjustments, for example, Glm 5.3 is way better than GPT Sol 6 in Hermes.