UCSD PhD student studying LLMs Ex-Meta AI, Berkeley AI Research
Joined August 2023
- Tweets34
- Following151
- Followers161
- Likes234
Exciting to see Mistral use our positional encoding method! 🙂🇫🇷
LeRoPE improves RoPE with learned frequencies, essentially for free; if you’re interested in learning more, check out our blogpost at petroskarypis.github.io/blog…, or reach out to me and @kpetro_research!
Today, we release the weights of a new and improved Le Chonk, advancing the open frontier for agentic coding and cybersecurity.
💻 Built to build: Le Chonk ranks #1 on @cognition’s FrontierCode among open models, and sets a new benchmark for Pareto efficiency on @artificialanalysis.
🛡 Built to defend: State-of-the-art in cyberdefense, put through the paces by the world's leading cyber vendors, including @CrowdStrike, @cloudflare, @rubrik, @zscaler, and others.
Built for sovereignty: Make it yours. Download the weights on @huggingface , customize it with Mistral Forge, and deploy it wherever you need it.
Read more from our team and customers 👇:
Sean O'Brien retweeted
"LeRoPE: Learnable RoPE Frequencies Improve Language Modeling"
LLMs use hand-picked RoPE frequencies to understand token positions, but this paper shows they work better when the frequencies are learned.
With just 32 extra parameters, LeRoPE beats RoPE across scales and discovers a dominant positional band around 2.2x the training context.
(1/n)
32 extra parameters → 3.4% saved pretraining compute
RoPE’s rotation frequencies matter, but they’re usually fixed at hand-picked values. In a new work with @kpetro_research, we let the network decide: LeRoPE learns the RoPE frequencies during pretraining.
Thread below!
(9/n)
Full paper here: arxiv.org/abs/2607.10134.
Huge thanks to collaborators @kpetro_research, @shresearchucsd, Rui Zhu and of course our advisor Julian McAuley (@McAuleyLabUCSD).
1
3
56
Sean O'Brien retweeted
What advantage to use, and when? Everyone's proposing new advantage functions for RL with LLMs but nobody knows why they work or fail.
We break this down and build FADE a self-adapting advantage to get +14% on LiveCodeBench v6 in 40% less steps.
Paper: arxiv.org/pdf/2607.01490
Sean O'Brien retweeted
Are AI models for music truly listening, or just good at guessing? This critical question is at the heart of the latest Best Paper Award winner at #ISMIR2025!
Huge congratulations to Yongyi Zang, Sean O'brien, Taylor Berg Kirkpatrick, Julian McAuley, and Zachary Novack for their paper, "Are you really listening? Boosting Perceptual Awareness in Music-QA Benchmarks."
They expose how current benchmarks can be solved without genuine audio perception—even by text-only models! Their new framework, RUListening, creates evaluations that force models to prove they're actually hearing the music. A vital step forward for robust AI evaluation.
Sean O'Brien retweeted
New paper showing that Contrastive Decoding (CD) works really well for reasoning tasks, e.g. +6 on GSM8K and +4 on HellaSwag compared to greedy. CD searches for strings that are more likely under a good model than a weak model, emphasizing the improvement from the better model.
Contrastive Decoding Improves Reasoning in Large Language Models
paper page: huggingface.co/papers/2309.0…
demonstrate that Contrastive Decoding -- a simple, computationally light, and training-free text generation method proposed by Li et al 2022 -- achieves large out-of-the-box improvements over greedy decoding on a variety of reasoning tasks. Originally shown to improve the perceived quality of long-form text generation, Contrastive Decoding searches for strings that maximize a weighted difference in likelihood between strong and weak models. We show that Contrastive Decoding leads LLaMA-65B to outperform LLaMA 2, GPT-3.5 and PaLM 2-L on the HellaSwag commonsense reasoning benchmark, and to outperform LLaMA 2, GPT-3.5 and PaLM-540B on the GSM8K math word reasoning benchmark, in addition to improvements on a collection of other tasks. Analysis suggests that Contrastive Decoding improves over existing methods by preventing some abstract reasoning errors, as well as by avoiding simpler modes such as copying sections of the input during chain-of-thought. Overall, Contrastive Decoding outperforms nucleus sampling for long-form generation and greedy decoding for reasoning tasks, making it a powerful general purpose method for generating text from language models.
Excited to announce my new paper! Check it out: arxiv.org/abs/2309.09117
TL;DR: we improve LM reasoning with only 3-5 lines of code and 3% extra compute. The method requires no training, scales well, and earlier work shows that humans prefer its longer generations.
(1/8)
Contrastive Decoding Improves Reasoning in Large Language Models
paper page: huggingface.co/papers/2309.0…
demonstrate that Contrastive Decoding -- a simple, computationally light, and training-free text generation method proposed by Li et al 2022 -- achieves large out-of-the-box improvements over greedy decoding on a variety of reasoning tasks. Originally shown to improve the perceived quality of long-form text generation, Contrastive Decoding searches for strings that maximize a weighted difference in likelihood between strong and weak models. We show that Contrastive Decoding leads LLaMA-65B to outperform LLaMA 2, GPT-3.5 and PaLM 2-L on the HellaSwag commonsense reasoning benchmark, and to outperform LLaMA 2, GPT-3.5 and PaLM-540B on the GSM8K math word reasoning benchmark, in addition to improvements on a collection of other tasks. Analysis suggests that Contrastive Decoding improves over existing methods by preventing some abstract reasoning errors, as well as by avoiding simpler modes such as copying sections of the input during chain-of-thought. Overall, Contrastive Decoding outperforms nucleus sampling for long-form generation and greedy decoding for reasoning tasks, making it a powerful general purpose method for generating text from language models.
There’s plenty more research to be done: in many ways our formulation is naive, and on some tasks (especially truthfulness) contrastive decoding can harm performance. I’ll be looking into overcoming these shortfalls; excited to see where the research leads!
(7/8)
1
5
755
Special thanks to Mike Lewis (@ml_perception) and @MetaAI for a great research residency!
(8/8)
8
687