agentictac retweeted
Frontier scale, built in Europe.
Nearly 4,000 NVIDIA Grace Blackwell Superchips powered the training of Mistral Large 4, now in public preview.
Congrats to the @MistralAI team 🙌
Meet Mistral Large 4, aka Le Chonk.
• 1T parameters, natively multimodal. 49B active.
It is the best open weights model from US or Europe on aggregated benchmarks.
• State-of-the-art on critical workloads, including cyber defense, manufacturing and finance and it surpasses closed frontier models on visual grounding.
• Forged in Europe end-to-end and is deployable from Europe via our own Mistral Cloud infrastructure.
• Available to all via API today. Working with cybersecurity partners privately.
Open weights release end of October.
agentictac retweeted
Meet Mistral Large 4, aka Le Chonk.
• 1T parameters, natively multimodal. 49B active.
It is the best open weights model from US or Europe on aggregated benchmarks.
• State-of-the-art on critical workloads, including cyber defense, manufacturing and finance and it surpasses closed frontier models on visual grounding.
• Forged in Europe end-to-end and is deployable from Europe via our own Mistral Cloud infrastructure.
• Available to all via API today. Working with cybersecurity partners privately.
Open weights release end of October.
🤖 Made with AI
agentictac retweeted
really really like this
very elegant, based and theorypilled
lots of interesting open Qs around agentic extensions, e.g is turn-level resampling sufficient or do you need environment replay
but a rare principled free lunch for the awkward in-between zone we call “midtraining”
agentictac retweeted
Thrilled to share our latest research on verifying CoT reasonings, completed during my recent internship at FAIR @metaai. In this work, we introduce Circuit-based Reasoning Verification (CRV), a new white-box method to analyse and verify how LLMs reason, step-by-step.
agentictac retweeted
The paper makes vision language models reason better by sampling, learning, and scoring hidden chains of thought.
A 7B model beats GRPO by 10.6%.
It treats reasoning steps as hidden choices and trains a sampler to propose many explanations.
The sampler learns a distribution over good paths with a rule that spreads credit across steps.
For token level guidance it scores partial sequences and linearly interpolates gaps.
For safe exploration it keeps only samples that beat a strong reference and drops weak ones.
At test time it samples several rationales and answers, then picks the answer with the highest length adjusted probability.
This Bayesian rule, BiN, beats Best of N and avoids an external judge.
On visual math and general tasks, accuracy and reasoning diversity rise, with gains on diagrams and text reading.
The core idea is to collect many good reasoning paths and choose with a principled rule.
----
Paper – arxiv. org/abs/2510.23925
Paper Title: "Latent Chain-of-Thought for Visual Reasoning"
agentictac retweeted
Your Base Model is Smarter Than You Think
This new paper reveals how pure, inference-time sampling can unlock advanced reasoning from LLMs. It matches or outperforms reinforcement learning on complex tasks, all without any additional training or verifiers.
agentictac retweeted
Diffusion language models indeed have a better learning potential than AR models. We empirically verified that at scale: arxiv.org/abs/2511.03276
And I think the most challenging problem to build a DLM reasoner is the DLM RL algorithm; it has a large efficiency and performance gap compared to that of AR ones. We're working on that at: github.com/JinjieNi/MegaDLMs
DLMs or its similar variants will have a bright future!
agentictac retweeted
✨ New course materials: Interpretability of LLMs✨
This semester I'm teaching an active-learning grad course at @TelAvivUni on LLM interpretability, co-developed with my student @dhgottesman.
We're releasing the materials as we go, so they can serve as a resource for anyone curious about how LLMs work from the inside:
github.com/mega002/llm-inter…
agentictac retweeted
cool paper studying learning dynamics of RL. alphaxiv.org/abs/2511.08567v…
among other things:
- RL tends to produce sparse weight updates
- Different RL runs preserve a “narrow, stable set of parameters” that is invariant across runs
agentictac retweeted
Lots of cool papers from today's arXiv. This one provides evidence that all trained models share a low-rank subspace in their weights. They analyzed hundreds of models to come up with their figures. Cool work!
🔗arxiv.org/abs/2512.05117
agentictac retweeted
TurboDiffusion: 100–205× faster video generation on a single RTX 5090 🚀
Only takes 1.8s to generate a high-quality 5-second video.
The key to both high speed and high quality?
😍SageAttention + Sparse-Linear Attention (SLA) + rCM
Github: github.com/thu-ml/TurboDiffu…
Technical Report: jt-zhang.github.io/files/Tur…
agentictac retweeted
Interested in how @Kimi_Moonshot 's kimi linear attention (KDA) "improves" linear attention, I break down the math to show how it evolves all the way from the most basic version.
Linear attention can be seen in two perspectives:
- On the one hand the linear "fast memory" matrix is a sum of all value vectors in the context weighted by the key vectors, so when it is multiplied by the query vector, the outcome is just a non-softmax version of attention computed linearly.
- On the other hand the same construction of the fast memory can be seen as the gradient descent of a particular loss function. This is where people can choose different losses to make fast memory more powerful.
The choice of loss function to optimize:
- Naively the construction is equivalent to optimizing the correlation (inner product) between the latest key vector and the latest value vector. This means the fast memory will help the query vector "find" the right value vector given that it matches the right key vector.
- A step forward is optimizing the "reconstruction" (MSE) between the key and the value. This means the query vector's search for the right value vector can be even more accurate. This is DeltaNet.
- The most recent step is to add a scalar to the old fast memory so that as we optimize the search for the new value, we gradually "forget" the influence of previous values in the attention. This is Gated DeltaNet.
The innovation of KDA:
- A scalar value will demolish the influence of previous key-value pairs across all dimensions, but we want a fine-grained gate to control the forgetting each dimension.
- So KDA introduces a diagonal matrix gate for each head, so that each dimension of the head will be assigned a forgetting scalar.
- This diagonal gate is built from the linear combination of a token-dependent gate (g) to control "how much to forget in this dim for this token) and a token-independent gate (A) to control "how much to forget in this dim in general).
In reality they have done some optimizations to make the inference much faster, such as reformatting the update rule in chunks and materializing the diagonal matrix as a vector.
But the core math is just to make surgical choices to forget previous information along semantically meaningful heads/dimensions. Pretty cool!
agentictac retweeted
If anyone asks for the best AI paper of the week, it’s definitely this elegant work: arxiv.org/pdf/2512.16922
agentictac retweeted
Can we understand the chain-of-thought (CoT) of latent reasoning LLMs using current mech interp techniques?
It turns out we can uncover interpretable structure, at least on simple math problems!
In a short study we show that latent vectors represent eg. intermediate calculations
agentictac retweeted
We (@lawrence_cjs, @yuyangzhao_ , @shanasaimoe) from the SANA team just posted a blog on the core of Linear Attention: how it achieves infinite context lengths with global awareness but constant memory usage! We explore state accumulation mechanics, the evolution from Softmax to Linear KV caches, applications in long video generation (like our SANA-Video), and advanced strategies for LLMs to overcome retrieval limits. Perfect for scalable AI! Read more: hanlab.mit.edu/blog/infinite…
The training/ Inference code and checkpoints are released. Welcome to try! github.com/NVlabs/Sana
agentictac retweeted
Much like the switch in 2025 from language models to reasoning models, we think 2026 will be all about the switch to Recursive Language Models (RLMs).
It turns out that models can be far more powerful if you allow them to treat *their own prompts* as an object in an external environment, which they understand and manipulate by writing code that invokes LLMs!
Our full paper on RLMs is now available—with much more expansive experiments compared to our initial blogpost from October 2025!
arxiv.org/pdf/2512.24601
