@Keyuciallo

MSc @uni_tue; RA @MPI_IS

 Tübingen, Deutschland
keyu-wang-2002.github.io/
Joined November 2025
  • Tweets32
  • Following36
  • Followers210
21Media
🔥 Frontier labs are redesigning the residual stream to beat the curse of depth: Kimi K3 uses AttnRes, DeepSeek V4 uses mHC, ByteDance proposed HC, and there are more: LNS, KEEL, MoDA… 🙋 But each method was tested with its own training budget, model design and codebase, so the results can't be compared directly. Which one actually works best? More importantly, which one turns architectural depth into effective computational depth? 👇 We built DepthBench to find out. 📄 arxiv.org/abs/2609.32534
5
41
12
328
60,812
DeepSeek
@deepseek_ai19h
🚀 Introducing DeepInfer, our in-house inference framework, now fully open source! Powering DeepSeek-V4.1-Flash at scale with high throughput and low latency. We’re sharing our production stack to help the community lower compute costs, reduce energy use, and make AI more affordable. 🔗 github.com/deepseek-ai/DeepI… 1/4

GitHub - deepseek-ai/DeepInfer: DeepInfer: High-performance, large-scale inference framework for...

DeepInfer: High-performance, large-scale inference framework for DeepSeek models - deepseek-ai/DeepInfer

github.com
1
3
146
Keyu Wang retweeted
Philipp Nazari@philna0019h
Great summary of our paper, Depth Memory! We 1. introduce a framework for existing depth mixing methods and 2. propose a new depth mixer that outperforms AttnRes and mHC. Check it out! 🚀 @_davidgonmar @tk_rusch @Shiwei_Liu66
alphaXiv
@askalphaxiv20h
"Depth Memory" Attention Residuals stores earlier Transformer layer outputs for retrieval but this memory grows with depth. Linear Depth Memory (LDM) maintains a fixed-size memory across layers, using recursive least squares to store and retrieve earlier layer information alongside the standard residual stream. On 400M and 1.5B models, LDM achieves lower perplexity than standard residuals, mHC, and AttnRes, while maintaining competitive zero-shot accuracy and comparable throughput with optimized kernels. alphaxiv.org/abs/2610.18809
5
15
687
Keyu Wang retweeted
Wanqi Yang@WanqiYang1Oct 19
Cognitive debt may be more dangerous in the AI era. Introducing TraceVista, a tool for understanding agent trajectories, built around how people naturally make sense of complex work. Try & examples: tracevista.pages.dev/ Code & skill: github.com/Superone77/TraceV…
12
22
4
189
10,770
Keyu Wang retweeted
ethantsliu
@ethantsliuOct 16
should expand it to a "zoo" of pretraining and then a "zoo" of post-training 😆
Keyu Wang
@KeyucialloOct 15
mHC, iHC, oHC, xHC, and a zoo of AttnRes variants... An LLM architect with FOMO from the daily explosion of new papers? 🫠 Fear not! You can't afford to miss ScalingDepth — our community hub for keeping up with the ever-evolving world of residual connections! We'll keep tracking and sharing the latest research, methods, and papers from the community 🚀 And it's an OPEN community! Missed something? PRs and DMs are always welcome! Got a cool blog, post, idea, or unexpected finding? We'd love to hear it! ScalingDepth hosts a dedicated space to showcase community insights and discoveries.🌍 Built by the community. Driven by curiosity. Pushing the limits of depth! 🔥 🌐 keyu-wang-2002.github.io/Sca…
1
4
360
Keyu Wang retweeted
Xiangdong Zhang
@aHapBeanOct 15
A great resource for keeping track of residual connection research. Love the space for community insights alongside the papers! We’re also preparing the xHC code release and look forward to sharing it with everyone soon! 🚀 Feel free to star the repo for updates! github.com/aHapBean/xHC And we’ve got more cooking 👀

GitHub - aHapBean/xHC: [Tech Report, NeurIPS 2026] Expanded Hyper-Connections

[Tech Report, NeurIPS 2026] Expanded Hyper-Connections - aHapBean/xHC

github.com
Keyu Wang
@KeyucialloOct 15
mHC, iHC, oHC, xHC, and a zoo of AttnRes variants... An LLM architect with FOMO from the daily explosion of new papers? 🫠 Fear not! You can't afford to miss ScalingDepth — our community hub for keeping up with the ever-evolving world of residual connections! We'll keep tracking and sharing the latest research, methods, and papers from the community 🚀 And it's an OPEN community! Missed something? PRs and DMs are always welcome! Got a cool blog, post, idea, or unexpected finding? We'd love to hear it! ScalingDepth hosts a dedicated space to showcase community insights and discoveries.🌍 Built by the community. Driven by curiosity. Pushing the limits of depth! 🔥 🌐 keyu-wang-2002.github.io/Sca…
3
16
892
Keyu Wang retweeted
J.L@Jo1uckOct 15
Memory Attention paper update! Added extensive ablation studies exploring MA integration, placement, normalization, and Attnres. simple idea: V = K + M. More parameters, yet a lower GPU memory footprint, better performance, and no V cache! 💻 joluck.github.io/memory-atte…
J.L@Jo1uckSep 24
Could this be a direction for future model architectures? Introducing Memory Attention (MA). alphaxiv.org/abs/2609.28399
5
13
461
mHC, iHC, oHC, xHC, and a zoo of AttnRes variants... An LLM architect with FOMO from the daily explosion of new papers? 🫠 Fear not! You can't afford to miss ScalingDepth — our community hub for keeping up with the ever-evolving world of residual connections! We'll keep tracking and sharing the latest research, methods, and papers from the community 🚀 And it's an OPEN community! Missed something? PRs and DMs are always welcome! Got a cool blog, post, idea, or unexpected finding? We'd love to hear it! ScalingDepth hosts a dedicated space to showcase community insights and discoveries.🌍 Built by the community. Driven by curiosity. Pushing the limits of depth! 🔥 🌐 keyu-wang-2002.github.io/Sca…
1
23
5
137
7,232
more replies
4/5 Too many papers, too little time? We've got you! Explore research on residual connections, normalization, and depth scaling — all in one place. Stay curious. Stay updated. Beat the FOMO! 🚀
1
1
3
169
5/5 💡 Got a crazy idea? An unexpected result? An insight that didn't make it into your paper? Don't let it die in your notes! ScalingDepth welcomes blogs, research insights, and fresh perspectives from the community. Your ideas deserve to be seen! 🔥 Website: keyu-wang-2002.github.io/Sca…
1
3
158
super cool to see that Linear Depth Memory achieves impressive performance on DepthBench @_davidgonmar.
David González-Martínez@_davidgonmarOct 14
mHC and AttnRes demonstrate that richer depth mixing improves model performance at scale. Is there a unifying perspective? 🤔 Meet our Depth Memory framework: each layer reads from a set of streams to form its input, then writes back its output. Linear Depth Memory (LDM) equips models with a matrix-valued memory.
1
5
25
2,269
surprisingly, LDM is still not SOTA at extreme depth, which makes me even more convinced that depth scaling is a challenging yet promising direction!
1
1
277
Love this sweet illustration!
Sophie Wang
@SophieLWangOct 12
I wrote a blog post detailing the writing approach I took while working on my recent paper: sophielwang.com/blog/paper-w… I describe the behind the scenes for how this paper came to be, why and how I storyboarded figures before running experiments, and some figure design tips! :)
4
452
[10/11] So the bigger picture is: 🚀 Depth is an underexplored and promising scaling axis, but only with the correct residual connections, and system-level design are badly needs to fulfill it in practice. 📄 Paper: arxiv.org/pdf/2609.32534 💻 Code: github.com/keyu-wang-2002/De… 🤗 Models: huggingface.co/aspect-ratio-…
1
3
18
902
[11/11] DepthBench wouldn't exist without this team ❤️ @yangyi_huang_cn , @Jo1uck , @_davidgonmar, @Besteuler , and @Shiwei_Liu66
3
16
955
Load more