Veselin Raychev retweeted
Introducing a new paper, my first with the SRI Lab, where I have now officially started my PhD!
Lean proving agents are expensive. We came up with a method to efficiently query these agents, maintaining performance while decreasing query cost by 25.8%.
Veselin Raychev retweeted
🚀 INSAIT is pleased to unveil the next generation of BgGPT! The third version of the Bulgarian AI chat can now perform image and speech recognition, process documents and search the internet.
📱 The chat is freely accessible to all Bulgarian citizens via bggpt.ai, as well as through the new mobile iOS and Android apps. It supports longer conversations and can process large documents thanks to the nearly 20x larger context window.
🖼️ In addition to the public chat, INSAIT now releases the models behind BgGPT 3.0 in three sizes – 4B, 12B, and 27B. They are based on Google’s Gemma 3 and have been continually trained using novel research methods developed at INSAIT on over 100 billion Bulgarian words. This allows BgGPT 3.0 27B to outperform nearly 10x larger models on Bulgarian tasks. The models can be freely downloaded and integrated by any company or institution and can be directly used via an API key. For access and more information, visit: models.bggpt.ai
🛡️ The entire BgGPT platform built by INSAIT, including the LLMs, is of key strategic importance for Bulgaria’s technological AI sovereignty and data protection. Complex joint projects already launched by INSAIT with institutions such as the National Revenue Agency and the National Audit Office to integrate BgGPT, can now be carried out on proprietary infrastructure, guaranteeing security, confidentiality, and independence from external parties.
Veselin Raychev retweeted
Coding agents are now frequently used to maintain software. A bug report comes in, the agent writes a patch, and submits it. But what happens when the bug is already resolved? 🙃
It turns out even SOTA models apply another fix over 50% of the time. More in our blog post.👇
Today, we’re excited to announce Snyk Agent Security and the general availability of Snyk Evo AI-SPM. 🚀
You cannot slow down AI coding agents, but you cannot let them bypass your security stack either. It’s a shadow AI crisis. 👾
See the fix here : evo.ai.snyk.io/
I am guessing minimax 2.5 must have distilled Sonnet 4.5.
Our new SWE-bench Multilingual leaderboard compares software engineering performance across 9 different languages as evaluated with mini-SWE-agent v2. Model rankings are significantly different between languages. Detailed stats & browsable trajectories in 🧵
1
1
174
Veselin Raychev retweeted
Are AGENTS actually useful for coding agents?
In our latest preprint, we analyze the effect of context files on coding model performance.
Short TLDR: It depends. Manually-written files help, while LLM-generated files (e.g. by the agent) don't.
More details in the thread 🧵
Veselin Raychev retweeted
1/🧵 LLMs can write their own benchmarks to uncover security vulnerabilities!
We leverage LLMs to expand BaxBench with 40 entirely novel, complex web backend tasks, more than doubling the original benchmark, resulting in AutoBaxBench. These tasks include extensive test cases and end-to-end exploits to expose vulnerabilities in implementations, which we confirm match or even outperform human-written exploits.
How we made SWE bench containers only 5GB down from over 240 originally. Work by Mark Müller and Christian Mürtz from @logic_star_ai and @the_sri_lab. logicstar.ai/blog/how-we-mad…
2
103
Veselin Raychev retweeted
🚨 AI agents wrote 7% of all GitHub PRs in June. But can we trust their code?
We built Agents in the Wild – a live dashboard tracking autonomous AI agents across GitHub to answer that question: insights.logicstar.ai/
Here’s what we learned from analyzing 10M+ PRs 👇 1/n 🧵
Veselin Raychev retweeted
If you are at ICLR, come by our posters in the DL4C (Garnet 218-219) and BuildingTrust (Hall 4 #6) workshops.
We have now evaluated over 30 models, including Grok 3, Gemini Pro, and o3—none of them are ready for you to give in for pure vibes in coding.
baxbench.com
Veselin Raychev retweeted
🚀 We are delighted to announce MamayLM, a new state-of-the-art efficient Ukrainian LLM!
📈 MamayLM surpasses all similar-sized models in both English and Ukrainian, while matching or overtaking up to 10x larger models.
📊 MamayLM is a 9B model that can run on a single GPU, enabling cost-efficient AI autonomy and adoption across sectors in Ukraine such as education, legal, healthcare, public services and others (e.g., by specializing it to particular use cases). MalayLM is also attractive for organizations wishing to preserve data privacy as it s efficiency allows it to run on a local machine.
🧠 MamayLM is trained on high-quality Ukrainian data and understands Ukrainian language, culture, and history. It is built on top of Google’s Gemma 2 9B model, but uses a number of new advances stemming from INSAIT’s experience in creating BgGPT, a Bulgarian LLM we released last year, now adopted nationwide and profiled several times by Google as a worldwide success case.
🤝 MamayLM is developed in a collaboration between researchers at INSAIT and ETH Zürich (@ETH_en) and is trained entirely via donations to INSAIT for AI compute resources.
📥 MamayLM is now freely available to download on INSAIT’s HuggingFace in both full and quantized versions. We also publicly release all Ukrainian benchmarks we evaluated on.
📝 Further, we release blog posts in both English and Ukrainian, sharing our approach to creating MamayLM, hoping to drive further improvements by the community.
🌎 The release of LLMs for various languages is part of INSAIT’s mission in ensuring countries can achieve AI autonomy in a cost-efficient, controlled, safe and predictable manner.
🔗 In the comments: a link to the model and the blog post about it (in English and Ukrainian)!
#MamayLM #INSAIT #ETHZurich #UkrainianAI #LLM #OpenSourceAI #GovTech #EduTech #HealthcareAI #NLP
Veselin Raychev retweeted
🚨 New SWT-Bench Submission! 🤖
Amazon Q Developer Agent leads the SWT-Bench leaderboard 🥇 with an impressive 49% of successfully tested issues and a coverage improvement of 57% on SWT-Bench Verified.
Veselin Raychev retweeted
Say hello to Europe's six new new AI Factories:
🇦🇹 AI:AT
🇧🇬 BRAIN++
🇫🇷 AI2F
🇩🇪 JAIF
🇵🇱 PIAST
🇸🇮 SLAIF
These factories will help startup, small and medium companies develop AI models using top supercomputing power, data, and talent ↓
Veselin Raychev retweeted
Claude 3.7 is making waves, being featured in impressive demos and showing strong results on common benchmarks. But can it generate correct and secure critical code modules? 🧵
If you build backends and know interesting things about securing them, you can contribute to BaxBench and get your scenarios added.
github.com/logic-star-ai/bax…
79
The next frontier in AI: autonomous app creation! BaxBench evaluates LLMs on 392 app backend creation tasks, highlighting the gap between correctness and security. @the_sri_lab @logic_star_ai
baxbench.com
While top models like o1 are great for prototyping and produce 60% “functional” code, we could execute security exploits in around half of the cases.
@mark_veroe @ni_jovanovic @nielstron @tch1bo
3
122























