Ethan Dyer retweeted
BioMysteryBench, our new bioinformatics eval, tests whether Claude can devise creative solutions to open-ended research problems.
Read more: anthropic.com/research/Evalu…
Ethan Dyer retweeted
A statement on the comments from Secretary of War Pete Hegseth.
anthropic.com/news/statement…
Ethan Dyer retweeted
Opus 4.6 is our most capable Computer Use model to date. Excited for everyone to give Computer Use a try with Claude in Chrome, Cowork, and Claude Code!
To celebrate, I let Claude (4.6) Monet show off his artistic side in the Claude for Chrome extension.
Ethan Dyer retweeted
I'm excited about this! Our team has been working really hard to improve Gemini 1.5 capabilities significantly on multiple fronts and in particular MATH/STEM! Please see the report here:
goo.gle/GeminiV1-5
Today we have published our updated Gemini 1.5 Model Technical Report. As @JeffDean highlights, we have made significant progress in Gemini 1.5 Pro across all key benchmarks; TL;DR: 1.5 Pro > 1.0 Ultra, 1.5 Flash (our fastest model) ~= 1.0 Ultra.
As a math undergrad, our drastic results in mathematics are particularly exciting to me!
In section 7 of the tech report, we present new results on a math-specialised variant of Gemini 1.5 Pro which performs strongly on competition-level math problems, including a breakthrough performance of 91.1% on Hendryck’s MATH benchmark without tool-use (examples below 🧵).
Gemini 1.5 is widely available, try it out for free here aistudio.google.com & read the full tech report here: goo.gle/GeminiV1-5
Ethan Dyer retweeted
Today we have published our updated Gemini 1.5 Model Technical Report. As @JeffDean highlights, we have made significant progress in Gemini 1.5 Pro across all key benchmarks; TL;DR: 1.5 Pro > 1.0 Ultra, 1.5 Flash (our fastest model) ~= 1.0 Ultra.
As a math undergrad, our drastic results in mathematics are particularly exciting to me!
In section 7 of the tech report, we present new results on a math-specialised variant of Gemini 1.5 Pro which performs strongly on competition-level math problems, including a breakthrough performance of 91.1% on Hendryck’s MATH benchmark without tool-use (examples below 🧵).
Gemini 1.5 is widely available, try it out for free here aistudio.google.com & read the full tech report here: goo.gle/GeminiV1-5
Ethan Dyer retweeted
In writing this paper, there were countless features we thought might be bugs. After careful inspection, ~all of them revealed surprising and subtle model properties.
To me this capacity for surprise is the true test of a new technique.
This thread is about my favorite finding.
Ethan Dyer retweeted
Nature research paper: Universality in long-distance geometry and quantum complexity go.nature.com/3ZFVmuu
Ethan Dyer retweeted
Excited to announce that the entire Blueshift team has joined @DeepMind! We will be working with @OriolVinyalsML and others to advance capabilities of LLMs developed by DM / Alphabet! We hope to continue to grow DM's presence in Bay Area and New York in the coming months :-)
Ethan Dyer retweeted
If you are interested in solving challenging multi-step reasoning problems with LLMs, join us!
We have an opening for a Research Scientist position at Blueshift!
Learn more about the role & apply here:
forms.gle/VZ8oHsuswt3iXwqf8
Learn about our team:
research.google/teams/bluesh…
Very excited to present Minerva🦉: a language model capable of solving mathematical questions using step-by-step natural language reasoning.
Combining scale, data and others dramatically improves performance on the STEM benchmarks MATH and MMLU-STEM. goo.gle/3yGpTN7
1/ Super excited to introduce #Minerva 🦉(goo.gle/3yGpTN7). Minerva was trained on math and science found on the web and can solve many multi-step quantitative reasoning problems.
Very excited to present Minerva🦉: a language model capable of solving mathematical questions using step-by-step natural language reasoning.
Combining scale, data and others dramatically improves performance on the STEM benchmarks MATH and MMLU-STEM. goo.gle/3yGpTN7
2/ Among many impressive properties, one side effect of training on the web is that Minerva has seen text used to draw mathematical figures and so can sometimes reason about diagrams.
3/ Find out more about Minerva in the blog post (goo.gle/3yGpTN7), paper (arxiv.org/abs/2206.14858) or explore more minerva samples (minerva-demo.github.io)!
Ethan Dyer retweeted
Thrilled to announce🦉Minerva: a large language model capable of solving mathematical problems using step-by-step reasoning in natural language.
See blog here: goo.gle/3yGpTN7 and samples here: minerva-demo.github.io (1/n)
Very excited to present Minerva🦉: a language model capable of solving mathematical questions using step-by-step natural language reasoning.
Combining scale, data and others dramatically improves performance on the STEM benchmarks MATH and MMLU-STEM. goo.gle/3yGpTN7
Ethan Dyer retweeted
Very excited to announce a significant milestone in expanding reasoning capabilities of language models! 🎉🎉
We introduce #Minerva🦉: a language model that can solve mathematical questions using step-by-step natural language reasoning:
bit.ly/3OBj2d5
🧵
1/
Very excited to present Minerva🦉: a language model capable of solving mathematical questions using step-by-step natural language reasoning.
Combining scale, data and others dramatically improves performance on the STEM benchmarks MATH and MMLU-STEM. goo.gle/3yGpTN7
Ethan Dyer retweeted
Super excited to share Minerva!! – a language model that is capable of solving MATH with 50% success rate, which was predicted to happen in 2025 by Steinhardt et. al. (bounded-regret.ghost.io/ai-f…)!
#Minerva
1/
Very excited to present Minerva🦉: a language model capable of solving mathematical questions using step-by-step natural language reasoning.
Combining scale, data and others dramatically improves performance on the STEM benchmarks MATH and MMLU-STEM. goo.gle/3yGpTN7
Ethan Dyer retweeted
Very excited to present Minerva🦉: a language model capable of solving mathematical questions using step-by-step natural language reasoning.
Combining scale, data and others dramatically improves performance on the STEM benchmarks MATH and MMLU-STEM. goo.gle/3yGpTN7
Ethan Dyer retweeted
Language models have shown strong performance on a variety of #NLU tasks but are weaker at solving tasks that involve quantitative reasoning. Learn how #Minerva uses step-by-step reasoning to achieve a new state of the art on quantitative reasoning tasks→ goo.gle/3yGpTN7
Ethan Dyer retweeted
BIG-bench was the single biggest reason I joined @jaschasd's team at Google a year ago! Not only is it ground-breaking science—evaluating 200+ diverse tasks on 30+ LMs, it is also a revolutionary way of doing science: inviting the entirely community to join from the very start.
Ethan Dyer retweeted
You may want to check out the Symbol-Interpretation task in Big-Bench [f1].
PaLM and other LLMs do not go further than random-chance (20%), even >0-shot.
Avg human scores 36%, best human 100%. [f2-3]
Ethan Dyer retweeted
Google has released the 442 author 132 institution extremely diverse "BIG-Bench" Neural Scaling Laws Benchmark Evaluation Paper.
"Beyond The Imitation Game: Quantifying And Extrapolating The Capabilities Of Language Models"
github.com/google/BIG-bench/…
