@HugoKoubbi
Joined February 2024
  • Tweets17
  • Following39
  • Followers19
8Media
Koubbi Hugo retweeted
Nirmit Joshi@nirmitj_12 Jun 2025
Happy to share our work (with amazing @HugoKoubbi , Theodor Misiakiewicz, Nati Srebro) We argue that spherical harmonics—rather than Hermite polynomials—provides a natural basis for this problem to obtain a more transparent picture. arxiv.org/abs/2506.09887

Learning single-index models via harmonic decomposition

We study the problem of learning single-index models, where the label $y \in \mathbb{R}$ depends on the input $\boldsymbol{x} \in \mathbb{R}^d$ only through an unknown one-dimensional projection...

arxiv.org
1
6
24
3,587
It's very difficult to improve the *exponent* in scaling laws for loss vs compute, especially by changing the optimizer! Our new paper shows that scaling momentum correctly can *provably* improve the scaling exponent on a theoretical model. Empirically, it works on LSTMs too!
12
64
10
309
56,631
🇮🇱 FLASH - "Je m’emmerde alors je tire" : des réservistes israéliens, après avoir combattu dans l'enclave palestinienne, dénoncent l’absence de règles d'engagement. Les soldats tirent à leur guise "sur tout ce qui bouge", détruisent des immeubles et laissent des rues jonchées de cadavres, avec l’approbation tacite de leurs supérieurs. (Courrier International)
440
4,296
278
9,941
1,217,647
Statistical optimal transport ift.tt/z4oy1XU

Statistical optimal transport

We present an introduction to the field of statistical optimal transport, based on lectures given at École d'Été de Probabilités de Saint-Flour XLIX.

arxiv.org
2
68
2
318
33,061
The Impact of LoRA on the Emergence of Clusters in Transformers ift.tt/zmaP02B

Understanding Catastrophic Forgetting In LoRA via Mean-Field...

Low-Rank Adaptation (LoRA) is the dominant parameter-efficient fine-tuning method due to its favorable compute-performance trade-off, yet it suffers from catastrophic forgetting. We study...

arxiv.org
1
17
99
11,016
If only the attention matrix V is changed, the tokens of the modified trajectory initially follow a similar pattern to the original dynamics before diverging towards a new clustering (Original dynamic on the left and LoRA dynamic on the right) 5/10
1
2
229
more replies
We also trained the architecture considered in the article on modular addition, and then fine-tuned it for modular subtraction. As an illustration of our results, the deeper the model, the faster the accuracy decreases during fine-tuning. (9/10)
1
176
Even if our hypotheses are rather simplistic, it seems that the clustering phenomenon also appears in LLMs such as Llama2 7B. (10/10)
1
163
Geskovski et al. [arxiv.org/abs/2305.05465,htt……] demonstrated that the dynamic asymptotically leads to the formation of token clusters. In this study, we investigate the impact of LoRA fine-tuning on cluster formation in Transformers (figures from github.com/borjanG/2023-tran……) 3/10
1
162
We establish short-term stability results for attention matrix parameters. Our findings reveal that even if long-term dynamics of tokens with the same initial conditions diverge, on a short-term scale, the two trajectories are close. 4/10
1
147
The LoRA algorithm seems to be the most widespread fine-tuning method for LLMs. LoRA reduces parameter count by employing low-rank matrix factorization on attention mechanisms. In arxiv.org/abs/2402.15415 w/ @m_boussard and @Louis_Hdez, we study the impact of LoRA on tokens 1/10
1
1
3
5
765
Based on the framework proposed by Sander et al. arxiv.org/abs/2110.11773, we have adopted a simplified Transformers architecture. In line with the concept of neural ODEs, we view the layers as time, and the tokens as an interacting particle system. 2/10
1
172
Load more