Praneeth (@0xViking) header
Praneeth (@0xViking) profile picture

Deep into Inference Engineering SDE 3 @ UBER as EXT ex DEFI Solutions Developer building @meetdooora

Bengaluru
Joined April 2018
Photo by Praneeth (@0xViking)Photo by Praneeth (@0xViking)Photo by Praneeth (@0xViking)Photo by Praneeth (@0xViking)Photo by Praneeth (@0xViking)Photo by Praneeth (@0xViking)Photo by Praneeth (@0xViking)Photo by Praneeth (@0xViking)Photo by Praneeth (@0xViking)Photo by Praneeth (@0xViking)
1/12 MHA vs MQA vs GQA Watched a few videos on attention today.. then worked through the cache math. Multi-head, multi-query and grouped-query attention change how much memory a model needs while it generates text. #buildinpublic
1
1
38
11/12 That is 48 mebibytes (MiB). One MiB is 1,048,576 bytes. It is 0.046875 gibibytes (GiB), or 0.047 rounded. One GiB is 1,024 MiB. This is just the KV cache. It does not include model weights, temporary calculations or runtime memory.
1
2
12/12 The cache count uses KV heads, not query heads. If KV heads go from 32 to 4, this cache gets 8 times smaller when everything else stays fixed. That does not mean the whole model uses 8 times less memory.. the weights are still there.
1
1/4 My first pull request got merged into @vllm_project today. it is a small fix in the documentation, But somewhere we have to start. github.com/vllm-project/vllm… #buildinpublic
1
70
3/4 The docs said otherwise. One page said the default is 4 GB. Another page said 0. A third page used GiB instead of GB. Note: 1 GiB is about 7% more than 1 GB.
1
1
16
4/4 I read the source code. The setting applies only when you set it. The value is read in GiB. So 4 means about 4.29 GB. Not 4 GB. My fix corrects the default and the unit in the docs.
1
9
1/10 Day 2 Today I learnt batching. We group work from a few requests to use the hardware better. Request = one user's input asking for an answer. Batch = a group processed together. Simple idea. But how we form that group changes a lot.
1
1
1
90
9/10 But now someone has to manage that changing group. The scheduler is the part deciding which requests get work next. Every round, it checks who's done and who can join. More moving parts.. we still can't add every waiting request if there's no capacity.
1
10
10/10 I started with "batch a few requests to use the GPU better." Then learnt the three ways of doing it.. and how a finished request can be replaced while the others keep running. Read up to continuous batching in the chapter today. That's where I stopped.
7