Computer Science PhD Student at Northeastern University Research Fellow @GoodfireAI
Boston, MA
Joined December 2014
- Tweets197
- Following526
- Followers530
- Likes1K
What in-context mechanisms do we find, other than copying?
The first one that stands out is the "identity rule". Here, the answer is the same as the question after eliminating a recognized "identity" from the question, like "ab=a".
@MengdiWang10 has seen this in LLMs too!
🚨 Discover the Science of LLM! We uncover how LLMs (Llama3-70B) achieve abstract reasoning through emergent symbolic mechanisms:
1️⃣ Symbol Abstraction Heads: Early layers convert input tokens into abstract variables based on their relationships.
2️⃣ Symbolic Induction Heads: Intermediate layers perform sequence induction over these abstract variables.
3️⃣ Retrieval Heads: Later layers predict the next token by retrieving the value associated with the predicted abstract variable.
Neural networks are starting to "think" symbolically! 🧠🔍
Paper: arxiv.org/abs/2502.20332
#LLMs #AI #SymbolicAI @TaylorWWebb @KaixuanHuang1
If you pick a random puzzle (try one here: algebra.baulab.info), you'll see there's often more than one way to understand context.
@catherineols & @NeelNanda5 found LLMs infer meaning by induction-style copying, and that happens here too. But there are many other strategies.
New paper walkthrough! @charles_irl and I read through, discuss and share intuitions for In-Context Learning and Induction Heads by @catherineols, @nelhage, me and @ch402. This is probably the most surprising paper I've been involved, a 🧵 of takeaways:
youtube.com/watch?v=dCkQQYwP…
Thanks for sharing - this is an interesting talk. His split of the top 10 risks of AI into categories was interesting - I like that the green list are all things that seem manageable and that one could work on as a researcher.
@gabriel_ilharco previously showed that task fine-tuning could be composed via vector algebra on whole model parameters. Can our function vectors also also be composed as vectors?
Yes! Function vector algebra does compose task-specific information well on many tasks.
Introducing task vectors!
A new way to steer models by doing arithmetic with model weights. Subtract to make models forget, add to make them learn
📜: arxiv.org/abs/2212.04089
🖥️: github.com/mlfoundations/tas…
We take inspiration from @nostalgebraist, @guy_dar1 and examine our function vectors in terms of model embedding space. When we decode FVs into vocabulary tokens, we find that function vectors often decode to words contained in the task's output space!
1/ Today we (@megamor2, @ankgup2, @JonathanBerant) published our paper "Analyzing Transformers in Embedding Space". We showed that *all* model parameters (FF and attention) can carry interpretable information.
paper: arxiv.org/abs/2209.02535
LLMs perform a wide array of functions (input-output mapping operations), including NLP tasks. If they contain a vector space for functions, it would allow us to begin decomposing what a function *is* in an LLM. We investigate functions in in-context learning (ICL) prompts.
LLMs represent words as vector embeddings. Do they represent *functions* as vectors too?
Yes! This has implications for how we think about “reasoning” in language models. New preprint w/ @millicent_li, @arnab_api, @amuuueller, @byron_c_wallace, @davidbau:
functions.baulab.info/
