Beyza Ermiş retweeted
turns out you can get frontier performance by having a swarm of tiny models working on the problem together.
great work from the research team on this one :)
cohere.com/blog/lessons-from…
Announcing new research on scaling multi-agent teams for long horizon software engineering tasks. 🤖🧩📈
Work led by @keenansamway, @mrdanieldsouza, Omar Darwiche Domingues, @mziizm, @yichern_tan, and @nouhadziri
Read the paper: cohere.link/tStRYJS
Check out the blog post: cohere.link/0TzgKoy
Beyza Ermiş retweeted
This is how we trained Soft-SVeRL! Highly recommended over just yoloing a scalar score. It has additional mathematical guarantees:
arxiv.org/abs/2605.28561
Replying to @adithya_s_k
2/ If you can verify, verify. If you can't, break it down.
Half the environments are plain pass/fail. The 925 workplace tasks become 5,125 yes/no checks, each graded on its own, most by a model given the expected answer. Half right counts as wrong.
Beyza Ermiş retweeted
He who controls the verifier controls the intelligence.
Great to see a consensus emerge about scaling verification using criteria decomposition and repeated-verification.
We found this to be especially important for co-training the generator and the verifier for recursive self-improvement.
arxiv.org/pdf/2605.28561
Scaling self-verification with DeepSeek V4 Flash beats Claude Fable 5 on Terminal-Bench 2.1, while being 11x cheaper 💰
As open-source models become more capable, they can now generate large numbers of high-quality candidate solutions and verify their own outputs at very low cost.
For example, we find that sampling just 5 solutions with DeepSeek V4 Flash and ranking them using the same model with LLM-as-a-Verifier can lead to a significant boost in accuracy (79% → 88%), outperforming closed frontier models on Terminal-Bench.
Try it out today: github.com/llm-as-a-verifier…
More on verification scaling in my previous post.
Beyza Ermiş retweeted
My talk is now available on YouTube. Thanks @Cohere_Labs for hosting me and making this public.
How can LLM agents exhibit reliability in dynamic environments? 🤖⚡🔄
@SharonYixuanLi discusses key strategies and challenges for reliable agent behavior.
Watch the full session: youtu.be/VfuCUgFmroI
Beyza Ermiş retweeted
Reasoning models are getting better at producing answers, but capability without self-knowledge hits a ceiling fast.
In our recent paper we start from a simple observation 💡: Confidence isn't one thing, before reasoning confidence is a forecast, after, it becomes verification.
Announcing new research on calibrating confidence in reasoning models 🎯⚖️, led by @ConorFinlay3, @joshuakurien_, @TheyCallMeMr_, @mziizm, and @beyzaermis.
📜Read the paper: arxiv.org/abs/2606.24281
Beyza Ermiş retweeted
1/ Reasoning models can verbalize how confident they are that their answers are correct.
The problem: these confidence estimates are often poorly calibrated. Models can be highly confident even when their answers are wrong.
Beyza Ermiş retweeted
Can an LLM know when not to trust its own answer? 🤔
Reasoning models are powerful but overconfident, and their verbalised confidence is generally unreliable.
We built CALIBER: an RL method that teaches a model to estimate its confidence twice, before and after reasoning.
Beyza Ermiş retweeted
Excited to share North Mini Code: Our first open-source coding model. It is small but very strong in agentic coding for its size🔥
I’m incredibly proud of the team — Shipping a model like this takes outstanding research, engineering, infrastructure, and collaboration.
Really glad I got to be part of this one. Huge credit to @TheyCallMeMr_ for leading the work, and grateful to the whole team!
1/ RLVR has driven big gains in math and code because many outputs admit reliable automatic checks: an answer matches the expected result, or a program passes tests.
But many real tasks are not like that. Code can be functionally correct but qualitatively terrible or a response may satisfy 4 syntactic constraints but fail 1 semantic constraint.
Beyza Ermiş retweeted
Our latest agentic reasoning model — fast, multilingual, multimodal and fully open source ♥️
Built to run efficiently, available for all.
Beyza Ermiş retweeted
Command A+ from @cohere is out now :) its our best model yet and its open source apache 2.0
Beyza Ermiş retweeted
Great to join forces with Reliant AI and to grow our Berlin team.
Healthcare and biopharma need trustworthy AI that can run on-prem, and that is able to ground it statements on the latest biomedical research. Our state-of-the-art context engine together with tech&datasets from Reliant, will make AI for pharma even better.
A major step forward for sovereign enterprise AI in healthcare and biopharma.
Cohere has acquired @reliant_ai 🇨🇦🇩🇪
Beyza Ermiş retweeted
Exciting news for the future of AI!
@cohere and Aleph Alpha are partnering to build a global sovereign AI platform. As demand for AI grows, we are accelerating the development of next-generation frontier models while upholding strong standards for data security and ethical AI.
🚀 Sovereign AI for the world.
Cohere & Aleph Alpha form transatlantic AI powerhouse anchored in Canada & Germany! Combining our global scale with European R&D excellence to build sovereign, enterprise-grade AI. Security, privacy & trust for businesses & governments worldwide. #SovereignAI #AIPartnership
Learn more: businesswire.com/news/home/2…
Image from left to right: Rolf Schumann, Schwarz Digits, Samuel Weinbach, Aleph Alpha, Aidan Gomez, Cohere, Minister Solomon, Canada, Minister Wildberger, Germany
Beyza Ermiş retweeted
Extremely proud to present ✨Tiny Aya ✨
Tiny 🤏 but mighty 💪3.35B parameter models, massively multilingual from the ground up 🌎🌍🌏,
built with immense care w.r.t language representation🤗
We had a blast building this! 💗 Have at it! 🎆
Beyza Ermiş retweeted
Very proud to share today we’re releasing Tiny Aya✨🤏: small enough to run on your phone, strong enough to support 70+ languages.
Proof that multilingual progress comes from intentional design --- and evaluation that measures balance, not just peaks 🕯️
So happy this is out in EACL Findings! 🥳 Loved being able to contribute a bit, big congrats to @_joestacey_ and all the authors! Really nice practical takeaways on robustness for closed-source LLM fine-tuning.
Proud to share this work led by @oliverjbolton 🎉 We introduce SimMerge, a practical way to make model merging more reliable at scale. Paper and results in the thread 👇
Sadly, this echoes what many authors and ACs have been feeling. The peer-review system is under serious strain. We can't keep pretending it's fine.
I am an AC for ICLR 2026. One of the papers in my batch was just withdrawn. The authors wrote a brief response, explaining why the reviewers failed at their job. I agree with most of their comments. The authors gave up. They are fed up. Just like many of us. I understand. We pretend the emperor has clothes, but he is naked.
Here is the final part of their withdrawal notice. I took the liberty to make it public, to highlight that what we are doing with AI conference reviews these last few years is, basically, madness.
---
Comment: We thank the reviewers for their time.
However, upon reading the reviews for our paper, it became immediately apparent that the four "reject" ratings are not based on good-faith academic disagreement, but on a critical failure to read the submitted paper.
The reviews are rife with demonstrably false claims that are directly contradicted by the text. The core justifications for rejection rely on asserting that key components are "missing" when they are explicitly detailed in the manuscript. Some specific examples are (and many are even fake claims).
Claim: Harder tasks like GSM8K are missing.
Fact: GSM8K results are in many tables, like Table 2 (Section 4.2) and Appendix G.
Claim: The method does not use per-layer ranks.
Fact: This is the entire point of our method. The reviewer clearly mistook our method for the baselines. (Section 2, Table 1).
Claim: The GP kernel is not specified.
Fact: It is specified in Appendix E (Table 6).
Claim: There is no ablation of the method's three stages.
Fact: Section 4.4 ("Ablation Study") and Appendix J are dedicated to this.
Reviewers have a fundamental responsibility to read and evaluate the work they are assigned. The nature of these errors is so fundamental, so systemic in overlooking explicit content, that it goes far beyond what "limited time" or "oversight" can explain. This work has gone through several rounds of revision over the last year. In earlier submissions, the paper usually received borderline or weak-accept scores.
Numerous signs strongly suggest that some reviewers are relying entirely on AI tools to automatically generate peer reviews, rather than fulfilling their fundamental responsibility of personally reading and evaluating manuscripts.
We strongly protest this.
This is a gross disrespect to the authors. It is a flagrant desecration of the reviewer's sacred duty. It fundamentally undermines the integrity of the entire peer-review process.
Given that the reviews are not based on the actual content of our paper, we have decided to withdraw the submission.
We leave this comment so that future readers of the OpenReview page are aware that the items described as "missing" are already present in the submitted manuscript. These negative reviews for this submission are factually unsound and do not reflect the content of the paper. We cannot and will not accept an assessment that is not based on the work we actually submitted.
Beyza Ermiş retweeted
First acceptance at @NeurIPSConf :)
Super thrilled to share that our work, ‘The Leaderboard Illusion’ got accepted at @NeurIPSConf 2025! 🎉
Huge congrats to all my coauthors!
And special shout out to @mziizm @sarahookr @beyzaermis @YiyangNan — learnt a lot from each of you 💙
It is critical for scientific integrity that we trust our measure of progress.
The @lmarena_ai has become the go-to evaluation for AI progress.
Our release today demonstrates the difficulty in maintaining fair evaluations on @lmarena_ai, despite best intentions.
Beyza Ermiş retweeted
Our scholar program is our big bet on a delicate balance: driving impactful research in ML and guiding the next generation of thinkers and researchers.
Looking forward to reviewing all the applications! 🎶




























