https://xitter.cf/t.co/WLa029HZ8w https://xitter.cf/t.co/gldmv0jpup
Joined December 2010
- Tweets1K
- Following1.5K
- Followers282
- Likes1.5K
There's an idea in Thinkertoys (@MichaelMichalko) that stuck with me: you only need one idea to really work in your lifetime. The ideas are already in you. You just haven't sat down long enough to pull them out.
You can afford 15mins:
luma.com/mkqs80e3
The little things that show how the Codex team cares. Much easier to see token usage vs. the competitor that shall not be named. It's right there, 2 clicks max. Thanks @thsottiaux & the Codex team
Hey @mstafford, i put HDIT online if you want to do a guided session anytime. sithankanna.com/hdit/
1/
Every LLM caches K and V during generation, but never Q.
The reason is one line of the attention equation.
4/
For the newest token you need: one new query, and all the past keys and values.
But the old keys and values don't change when you append a token. Appending token t+1 doesn't touch it.
So you save them once and reuse them. That saved memory is the KV cache.
5/
And that's why you never cache Q: the next prediction only uses the newest query. The old ones are never asked anything again.
I'm doing a short live walkthrough of the full derivation 15 mins.
luma.com/rep13imt
My notes on Why KV cache and not QKV cache. Explained using a simple 1-Layer Transformer. Using karpathy's setup (ignoring the Batch dimension)
drive.google.com/file/d/1Y6B…
Sithan Kanna retweeted
Listening to this just after doing HDIT in San Francisco is the way 🇺🇸
(For HDIT see this - buttondown.com/hdit - and talk to @sithankanna)
Something interesting: REINFORCE (RL) can be shown to be equivalent to Supervised Learning. I used a one-layer network + Single RL rollout to make to make connection simpler.
Here's the weight update overview:
👇🧵
Side by side: structurally identical. With binary reward R(τ) ∈ {0,1}: - R=1 → RL update = SL update (generated samples become ground truth) - R=0 → No update
