Tübingen, Deutschland
Joined November 2025
- Tweets36
- Following38
- Followers211
- Likes41
Keyu Wang retweeted
This next paper is a bit quirky, but here goes:
Introspection is hard to study in both humans and LLMs. But, we do have internal senses that are verifiable, like proprioception (the sense of where your arm is right now).
What would proprioception look like in language models?
As a digital analogue to the sense of position, a model could verbalize which expert activated in a certain layer for any previous token.
It turns out that LLMs can learn a range of such proprioception skills where they understand and report on their internals. We can calibrate and score these reports, and evaluate whether training for proprioception changes model behaviors.
Keyu Wang retweeted
Great summary of our paper, Depth Memory!
We
1. introduce a framework for existing depth mixing methods and
2. propose a new depth mixer that outperforms AttnRes and mHC.
Check it out! 🚀 @_davidgonmar @tk_rusch @Shiwei_Liu66
"Depth Memory"
Attention Residuals stores earlier Transformer layer outputs for retrieval but this memory grows with depth.
Linear Depth Memory (LDM) maintains a fixed-size memory across layers, using recursive least squares to store and retrieve earlier layer information alongside the standard residual stream.
On 400M and 1.5B models, LDM achieves lower perplexity than standard residuals, mHC, and AttnRes, while maintaining competitive zero-shot accuracy and comparable throughput with optimized kernels.
alphaxiv.org/abs/2610.18809
Keyu Wang retweeted
Cognitive debt may be more dangerous in the AI era.
Introducing TraceVista, a tool for understanding agent trajectories, built around how people naturally make sense of complex work.
Try & examples: tracevista.pages.dev/
Code & skill: github.com/Superone77/TraceV…
Keyu Wang retweeted
should expand it to a "zoo" of pretraining and then a "zoo" of post-training 😆
mHC, iHC, oHC, xHC, and a zoo of AttnRes variants... An LLM architect with FOMO from the daily explosion of new papers? 🫠
Fear not! You can't afford to miss ScalingDepth — our community hub for keeping up with the ever-evolving world of residual connections!
We'll keep tracking and sharing the latest research, methods, and papers from the community 🚀
And it's an OPEN community! Missed something? PRs and DMs are always welcome! Got a cool blog, post, idea, or unexpected finding? We'd love to hear it! ScalingDepth hosts a dedicated space to showcase community insights and discoveries.🌍
Built by the community. Driven by curiosity. Pushing the limits of depth! 🔥
🌐 keyu-wang-2002.github.io/Sca…
Keyu Wang retweeted
A great resource for keeping track of residual connection research. Love the space for community insights alongside the papers!
We’re also preparing the xHC code release and look forward to sharing it with everyone soon! 🚀
Feel free to star the repo for updates! github.com/aHapBean/xHC
And we’ve got more cooking 👀
mHC, iHC, oHC, xHC, and a zoo of AttnRes variants... An LLM architect with FOMO from the daily explosion of new papers? 🫠
Fear not! You can't afford to miss ScalingDepth — our community hub for keeping up with the ever-evolving world of residual connections!
We'll keep tracking and sharing the latest research, methods, and papers from the community 🚀
And it's an OPEN community! Missed something? PRs and DMs are always welcome! Got a cool blog, post, idea, or unexpected finding? We'd love to hear it! ScalingDepth hosts a dedicated space to showcase community insights and discoveries.🌍
Built by the community. Driven by curiosity. Pushing the limits of depth! 🔥
🌐 keyu-wang-2002.github.io/Sca…
Keyu Wang retweeted
Memory Attention paper update!
Added extensive ablation studies exploring MA integration, placement, normalization, and Attnres.
simple idea: V = K + M.
More parameters, yet a lower GPU memory footprint, better performance, and no V cache!
💻 joluck.github.io/memory-atte…
Could this be a direction for future model architectures?
Introducing Memory Attention (MA).
alphaxiv.org/abs/2609.28399
