Selim retweeted
TTIC student Selim Jerad (@selimjerad), along with collaborators Anej Svete, Jiaoda Li, and Ryan Cotterell, present "Disentangling the Expressivity of RoPE" at #COLM2026. Thursday, Oct. 8, 11:00am-1:00pm in Grand Ballroom #103: buff.ly/pq9Yojs
I’m heading to COLM in SF to present our latest work on the expressive power of RoPE 🪢
Come say hi to chat about transformer expressivity and the role of PEs!
Paper: arxiv.org/pdf/2608.11909
Incredibly grateful to be part of the adventure! Thanks Will for having me on the boat :)
Introducing the Language, Expressivity, Interpretability, and Formalism (LEIF) Lab at TTIC — it's a cool crew!
leif-lab-ttic.github.io/
Selim retweeted
Transformers are Inherently Succinct by Pascal Bergsträßer, Ryan Cotterell & Anthony Widjaja Lin is an interesting contribution.
Various work (Hahn 2020, Li & Cotterell 2025 i.a.) has shown that transformers are less powerful than RNNs … yet somehow they do so well in practice.
If you are interested in information retrieval go check this out! 🇰🇷
I will be in Seoul with @marcantallard this week for ICML to present ModernVBERT 🇰🇷
Our poster session on Wednesday from 10:30PM in Hall A (#4104). Come chat with us!
Selim retweeted
(re: OpenAI and Anthropic IPO news, a preprint)
Scaling up training reliably improves LLMs, but it also increases training and inference costs, leading to massive capital expenditure by AI firms. How can we understand what level of LLM scaling is justified economically? 🧵⬇️
Selim retweeted
Given access to a language model, can we extract an interpretable object like a DFA that captures which strings a language model is likely to generate?
Our new work on automata learning theory studies this question. To be presented at ##ICLR2026 🎉
Selim retweeted
🚀 Machine Learning Summer School Okinawa 2027
We will organize MLSS Okinawa in March 2027 at OIST!
Registration will open August 2026 via OpenReview.
Top lecturers, beautiful Okinawa, and cutting-edge ML discussions.
📅 Please mark your calendar!
oist.github.io/mlss2027okina…
#MachineLearning #MLSS #AI #DeepLearning
Selim retweeted
Inviting submissions to the first Workshop on Formal Languages and Neural Networks!
We welcome posters dicussing the formal expressivity, computational properties, and learning behavior of neural networks!
Call for posters: flann.cs.yale.edu/cfp.html
Deadline: February 12, 2026
Transformers seem proficient at understanding linguistic syntax, but how could they be doing this theoretically?
Our new work with @lambdaviking, @AnejSvete, Sophie Hao and Ryan Cotterell characterizes how looped- and padded-transformers can recognize context-free grammars! 🌳
Overall, we show that looped transformers can recognize general CFLs with some extra padding, and that unambiguous grammars require less resources for recognition. We encourage future work to investigate if our bounds can be tightened.