We provide transparent, in-depth, data-driven AI industry analysis to help businesses explore AI, machine learning and other emerging technology use cases
San Francisco
Joined December 2018
- Tweets172
- Following94
- Followers719
- Likes21.7K
It combines three benchmarks:
Browser tasks: 50 tasks, one attempt each.
Classification: 1,655 single-decision questions, such as security advisory severity, FDA recall class, paper topic and checking whether a browser task is done.
Bias: multiple-choice questions.
Across 57 models, three are tied at the top with 60: AutoJev-27B, d1-ultra and decider-12b. Newcomer d1-ultra has the best browser score at 74% and costs $0.026 per 1,000 decisions.
Full leaderboard and methodology: aimultiple.com/system-one. We’d love feedback on which models or tasks we should add next.
We’ve been testing “System One” models, also called decision models. Instead of generating text token by token, they return a typed decision with a probability for each option in a single pass. To compare them, we put together the AIM System One Index.
AIMultiple retweeted
We benchmarked Mistral’s new OCR across 300 documents in handwriting, printed media and printed text. OCR 3 is behind Gemini and others. With a 6% difference in a dataset of 300 documents, the difference is statistically significant.
AIMultiple retweeted
It reminds me of Grok 4 which aced every good benchmark with a holdout dataset like LiveCodeBench. AI influencers were impressed. Then we tested it and got disappointed.
In most cases, like the hallucination benchmark below, it failed to reach the top position.
AIMultiple retweeted
ChatGPT's new agent almost broke our benchmark. We'll soon need a harder test.
This benchmark is not based on a public dataset that is included in OpenAI's models. While we explain the task clearly, data is not public, therefore models not have access to it.
We are introducing AI LMC-Eval, a coding benchmark with 100 questions & tested on 7 leading LLMs.
LMC stands for Logic / Math Coding. We presented the LLM with high school level logic and math problems and instructed it to write Python to solve them.
Agentic AI is still mostly hype. We asked 5 AI agents to fetch a prices of a specific product from original sources and got only 20% of the results.
Should we try this with other agents?
research.aimultiple.com/ai-a…
#ArtificialIntelligence is a game-changing technology for businesses, but there are still many myths and unclear points about AI. Take a look at our article to learn about them.
research.aimultiple.com/ai-s…
#MachineLearning
AIMultiple retweeted
@AIMultiple says most companies today allocate nearly 50% of their QA budgets to test automation! Why is automating QA so important?
Learn about its impact for rapid product release cycle in our blog: hubs.la/Q01ygqCz0
#Calsoft #QA #QAAutomation #ProductEngineering
Digital transformation is integrating digital technologies into all aspects of a business to meet the market and changing business requirements.
Learn about why digital transformation matters and some use cases:
research.aimultiple.com/digi…
#digitaltransformation
Web scraping enables businesses to get a bulk list of their target audience’s email addresses. It reduces human errors in manually entering email addresses into a database and accelerates marketing processes.
To learn more, read our comprehensive article.
research.aimultiple.com/emai…
Psychological factors such as users’ sentiments regarding policy changes or new investments greatly influence how stock prices change.
In this article, we’ll explore how sentiment analysis can be applied to stock market forecasts. #StockMarkets
research.aimultiple.com/sent…