CS PhD student @uchicago, Gym owner, Chicago Rationality organizer, Like Rats, Hate Force, etc.
Chicago
Joined March 2009
- Tweets3.5K
- Following839
- Followers504
- Likes303
Todd Nief retweeted
Excited to share this work with @KihoPark_ and @victorveitch!
Can we change how images are compared without changing their embeddings?
In Anchor Divergences, we show how distributions over text anchors make this possible.
A short thread
Encoding semantic similarity into vector similarity is a basic building block of AI systems. When studying this, we usually just think of a single fixed notion of vector similarity (e.g., cossim). However, semantic similarity is fundamentally *contextual*---a fixed similarity notion shouldn't suffice! Turns out the information geometric lens both greatly clarifies this apparent contradiction and also yields a simple and effective technique for getting contextual semantics with vector search.
@_AkashKannan_ @KihoPark_
arxiv.org/abs/2610.06919
Turns out that subliminal learning is *not* a LoRA artifact after all. You just need way more data with more parameters. Makes sense tbh!
Todd Nief retweeted
I'll be presenting our paper on information geometry at ICML!
📌 Thu (July 9), 5 PM, Hall A, Poster #3301
📌 Fri (July 10), Hall C (Mech Interp Workshop)
Come say hi if you're at ICML! Happy to discuss our work or grab a coffee and chat :)
Interpreting and controlling internal representations should be based on how the model actually uses them!
Turns out: information geometry makes this precise. We show how, and use it to derive a (provably & empirically) robust strategy for steering.
arxiv.org/abs/2602.15293
An LLM can learn an *obsession* (cats, oak trees, Metallica) through finetuning only on sequences of numbers. This phenomenon is called subliminal learning.
Why does this happen? Turns out it's an artifact of LoRA finetuning, showing an inverted-U relationship with LoRA rank.
Takeaways: Models are very weird!
Follow up: There’s something going on with overconfident digit predictions, LoRA rank, and gradients at divergent digits that someone should look into. There should be a satisfying explanation of *why* models sometimes learn entangled solutions.
Joint work with @harveyiyun @Bartleby_Kamoi and @universeinanegg!
Check out the full paper here: arxiv.org/abs/2606.00831
Todd Nief retweeted
We trained an LLM trained on an LLM trained on a…🌀🌀🌀
If the original model is sycophantic or just 'weird', will those traits begin to amplify?
Yes! But amplification is rare and typically comes at the cost of coherence—except in the case of DPO where things get dicey
🧵
Softmax models (like LLMs) have curves
Interpreting and controlling internal representations should be based on how the model actually uses them!
Turns out: information geometry makes this precise. We show how, and use it to derive a (provably & empirically) robust strategy for steering.
arxiv.org/abs/2602.15293
Todd Nief retweeted
📖 ≠ 🧪 The Story is Not the Science.
Code is submitted but rarely executed during peer review--an issue likely to worsen with research agents.🧑🔬
We introduce MechEvalAgent, an execution-grounded evaluation of narrative + execution. Verify the science, not just the story.
1/n
Todd Nief retweeted
Lack of diversity in your LLM generation?
(also noted by Artificial Hivemind, best paper @NeurIPSConf)
Time to bring your base model back!
An inference-time, token-level collaboration between a base and an aligned model can optimize and control diversity and quality!
To conclude:
1. Dynamic weight grafting is a new technique that allows localization of finetuned model behavior to specific token positions and model components
13/n
Blog post: toddnief.com/articles/dynami…
Paper: arxiv.org/abs/2506.20746
Code: github.com/toddnief/dynamic-…
14/14
