Does AI need human-like cognition? UVa Physics Bachelor . Did some Philosophy of comparative religion at Uchicago. Now interpreting LLMs
Joined May 2014
- Tweets28
- Following144
- Followers20
- Likes6
Chupei W retweeted
How do we build sparsity into JEPA representations by design, while preserving task-relevant information?
Introducing Rectified LpJEPA, a JEPA architecture that learns sparse, non-negative, informative representations through principled distributional regularization. 📐
📄 Paper: arxiv.org/abs/2602.01456
💻 Code: github.com/YilunKuang/rectif…
📝 Blog: yilunkuang.github.io/blog/20…
(1/n)
all LLMs stumble on co-referenced updates → strong context interference.
We stream N updates to the same key and ask for the current value. With 46 concudxrrent keys, accuracy collapses across all tested SOTA
ICML 2025 workshop Paper: arxiv.org/abs/2506.08184
Could try it out in a flash. Just a 24MB dataset that ALL-SOTA LLMs fails, but humans could read about 99%+ of the answer in 1 minute. (cognitive sci proven 🥳) ICML25/Kimi AI internal bench huggingface.co/datasets/Cog2…
Super easy for human, Consistent failure mode across SOTA LLMs—GPT-5, Gemini/Sonnet/Grok4. All hallucinate the same pattern. This feels crucial for evaluation teams, controllability and AI safety. #AIsafety @janleike @ch402 @demishassabis @JeffDean @elicollins @mikeyk @kevinweil
all LLMs stumble on co-referenced updates → strong context interference.
We stream N updates to the same key and ask for the current value. With 46 concudxrrent keys, accuracy collapses across all tested SOTA
ICML 2025 workshop Paper: arxiv.org/abs/2506.08184
This test is Super easy for humans(Cog-sci 🧠proven), ALL SOTA LLMs fails and hallucinate in the same error pattern on this tracking task we all do daily. Might interest you.huggingface.co/datasets/Cog2… @vidhisha_b
@besanushi
@jubitaneja
@HeidyKhlaaf
@huashen218
@npew @nikitabier @lm_zheng @lmarena_ai This super easy for humans but hard for LLM test (All LLMs hallucinate) might interest you.
@NeelNanda5 We’re seeing a consistent failure mode across SOTA LLMs on our task—same pattern from GPT-2→GPT-5 and Gemini/Sonnet. What do you think? I’ll send the interpretability study follow-up next (in your MATC 9.0 application)
all LLMs stumble on co-referenced updates → strong context interference.
We stream N updates to the same key and ask for the current value. With 46 concudxrrent keys, accuracy collapses across all tested SOTA
ICML 2025 workshop Paper: arxiv.org/abs/2506.08184
huggingface.co/datasets/Cog2… could try it out in a flash. Just a 24MB dataset that ALL-LLMs fail, but humans could read about 99%+ of the answer in 1 minute. (cognitive science proven)
Minimal demo:
Key1: Value_1
Key1: Value_2
…
Key1: Value_N
Q: “What’s the current value of Key1?”
Expected:
Value_N.
Observed: models often return earlier Value_k as N grows. ALL SOTA model fails after extensive reasoning steps
Data: huggingface.co/datasets/Cog2…
PI-LLM-Bench tests top-down control in LLMs—humans ace it, models flop. Perfect for probing interference limits. @Yampeleg @omarsar0 @arankomatsuzaki @MFarajtabar @RuohongZhang
Minimal, reproducible failure where models should follow ‘last write wins’ (return the final update) but don’t—answers drift to earlier values @GaryMarcus @bindureddy