Bill retweeted
分享一份最近收集的 AI Infra 入门阅读清单，建议收藏，慢慢看
AI 已经可以生成很多很好的内容，解释概念、翻译论文、整理资料，都越来越方便。但在这个时代，能静下心来，把一篇好文章认真读完，把没懂的地方再琢磨一遍，我觉得更加难能可贵
这次整理的材料，主要集中在大模型推理这一块。从 KV Cache、显存管理，到请求调度、推理加速，再到怎么判断性能有没有真正变好
做工程的朋友，可以顺着往源码和实验里走。平时看 AI 产业、算力和存储的朋友，也可以先把原理读明白
读文章，挑值得反复读的。
1. 如果只选一篇，我会选 Inside vLLM。
Inside vLLM: Anatomy of a High-Throughput LLM Inference System，作者 Aleksa Gordić
aleksagordic.com/blog/vllm
把很多零散的概念串进了一个真实系统：一个请求进来，怎么排队，怎么分配 KV Cache，怎么执行，最后又怎么把结果送回来
PagedAttention、continuous batching、chunked prefill、prefix caching、投机解码、P/D 分离、多卡和性能测试，都能在这条线上找到位置。
这篇适合当主线，后面的材料用来补不懂的地方
2. 想弄懂 continuous batching，先看 Anyscale 这篇。
Continuous Batching 博客
anyscale.com/blog/continuous…
llm用户的回答有长有短，有的已经生成完了，有的还在继续。已经空出来的位置，能不能马上接下一个请求？
这篇用静态批处理和连续批处理的对比图，把这个问题讲得很直观。看完你会理解，GPU 的效率，也取决于请求怎么安排
建议重点看图和机制
3. NVIDIA 的两篇，一篇帮你建立全貌，一篇教你怎么看性能。
Mastering LLM Techniques: Inference Optimization
developer.nvidia.com/blog/ma…
适合先建立基本认识：处理输入的 prefill 和逐步生成输出的 decode 有什么区别，模型权重和 KV Cache 分别占什么空间，量化、批处理、多卡并行又各自解决什么问题
LLM Inference Benchmarking: Fundamental Concepts
developer.nvidia.com/blog/ll…
这一篇我也很推荐。用户等多久才看到第一个 token，后面生成得快不快，整个系统一秒能服务多少请求，是不同的问题。TTFT、ITL、TPOT 这些指标，要连着测试口径一起看
4. 想再往下算一层，读 kipply 的 Transformer Inference Arithmetic
Transformer Inference Arithmetic
这是一篇 2022 年的文章。它带你从计算量、内存容量、带宽和通信开销出发，估算推理为什么快、为什么慢，以及 batch size 改变后会发生什么
硬件和模型的具体例子有年代了, 作为 optional 选读
Bill retweeted
So I've got a ton of DMs about the future of software engineering and to answer everyone I'll write it here.
First of all, I don't really think that software engineering will be gone anytime soon. It will evolve. For those who has a head on their shoulders, it has already evolved. Others will fail to see it first but they eventually will.
So, the first thing that changed is code review. There will be none of it except for several high stake cases where proof of correctness is needed, such as banking, contracts, planes, bridges, skyscrapers, health, missions to other planets, etc
This means that if you are working as a shit coder (which is about 90% of coders), you will not be asked to write code anymore. And that's ok: a shit coder's code looked like shit anyways, so writing it by hand full time was an insult to humanity.
But that doesn't mean that shit coder will have less job. To the contrary: your manager will expect more code from you not less. Don't you really think your manager or his VP manager or VP's C-suite manager will start vibe coding themselves? This doesn't scale: there are not enough of them and they are too expensive.
The remaining 10% of software engineers, those who have already been working on this high stake software will use AI to write two kinds of code:
1. One off data transformation from one format into another verifiable format. In this case, they will validate the result not the code.
2. A screen-size readable code that they will actually read and fix by hand or iterate with AI until they line what they read.
Architects of such high stakes software will still design the dependency graphs of modules in the projects. They will indicate which modules will be given to shit coders and which ones to software engineers.
Architects of any high stake software project will still have to understand compter science foundations, cloud computing foundations, database foundations, internet protocol foundations, security foundations, maintainability foundations, CPU foundations, parallelism foundations, algorithm complexity, machine learning, etc.
UX people will still be responsible for UX decisions and will have to know ergonomics, user psychology, experiment design, etc.
So, it's an evolution not a disruption.
This is why I'm still writing fundamental textbooks and building a world's best self-education platform.
Knowledge of how things work is now even more important than ever, because the era if narrow specialization in software industry is gone. Everyone now can build a new Microsoft or Google in a matter of a month, alone, and the employer's expectation will be the same: "Tomorrow I need my own OS with blackjack and hookers. Do the necessary and ping me when it's done."
Bill retweeted
来个暴论+干货：A➗根本就不懂 Harness.
在harness上，和OpenAI对比起来，A➗更像一个营销公司。
Codex Desktop 完全领先 Claude Desktop 一个时代。
●codex desktop 的UI UX被所有同行学习，定义了agent GUI的设计规范
●扎实优雅的架构设计。真正的技术扎实和设计优雅。老早就大幅度重构，实现了统一 runtime host: V2 使用了 app-server 架构，全端统一使用一个runtime。对比之下，CC 架构到现在还是一个 session 一个 bun，天天堆代码屎山。而codex做到了，甚至你可以用自己的Client连接别人部署的 Codex App Server。
A➗还老吹自己是，Infra工程师特别多。我怎么感觉你根本就是营销公司？你的设计和架构在哪呢？
●各种Agent harness benchmark测试基本都是效果最好的一批，不管是Maka做的Terminal-Bench 测试还是Runta 的 Agent harness 测试，都是最强的那一批。
●在Astra上更是实现了异步工具调用与后训练，边执行边推理，实现了非阻塞调用，大幅度加快了执行，能明显感觉到现在执行时间短了。
●Codex老早就实载了服务端侧的上下文压缩，所以Codex虽然上下文要短，但是实际上效果表现却非常好。
在Astra上更是实现了异步工具调用与后训练，边执行边推理，实现了非阻塞调用，大幅度加快了执行，能明显感觉到现在执行时间短了。
●Responses API 做了 mid-turn steering，实现了动态交互调整，在任务进行中可以插入新指令或中间修改要求，直接把穿插修改放到了底层的 API层。
全球第一的 Computer Use 的的能力更是不用多说了，大家都吹爆了。
●Codex 还做了上下文管理的前沿探索，Context Windows 也已经实现在Codex里面了。用上下文窗口的滑动，而不是用压缩。这样子就通过可搜索历史主动换窗，让长期的信息能够沉淀，可以被召回。
●Codex的Memory目前也是越做越好了，基本上是目前实现里面最无感的一家。并且实现了dreaming机制。
●实装的 PTC / Code Mode 机制配合 astra 强大的后训练能力。让模型的能力大幅度跃升。是真正的Harness和模型相互配合的典范。
●Claude团队设计 MCP也是埋了不知道多少坑，根本没有借鉴历史很多协议的设计经验和精华。Plan mode也是Claude设计的，现在已经成了历史糟粕了。
