People are skeptical that teachers don't know their students can't read. Yes, teachers know students can't decode words. But teachers have been taught that words aren't necessary for "reading". Here's a teacher teaching kids to "read" by looking at pictures. In fact the word they're "reading" is covered up so the students can't even see it! And listen to how she describes it at the end of the video: "reading and analyzing text"

AIME I 2025: A Cautionary Tale About Math Benchmarks and Data Contamination AIME 2025 part I was conducted yesterday, and the scores of some language models are available here: matharena.ai thanks to @mbalunovic, @ni_jovanovic et al. I have to say I was impressed, as I predicted the smaller distilled models would crash and burn, but they actually scored at a reasonable 25-50%. That was surprising to me! Since these are new problems, not seen during training, right? I expected smaller models to barely score above 0%. It's really hard to believe that a 1.5B model can solve pre-math olympiad problems when it can't multiply 3-digit numbers. I was wrong, I guess. I then used openai's Deep Research to see if similar problems to those in AIME 2025 exist on the internet. And guess what? An identical problem to Q1 of AIME 2025 exists on Quora: quora.com/In-what-bases-b-do… I thought maybe it was just coincidence, and used Deep Research again on Problem 3. And guess what? A very similar question was on math.stackexchange: math.stackexchange.com/quest… Still skeptical, I used Deep Research on Problem 5, and a near identical problem appears again on math.stackexchange: math.stackexchange.com/quest… I haven't checked beyond that because the freaking p-value is too low already. Problems near identical to the test set can be found online. So, what--if anything--does this imply for Math benchmarks? And what does it imply for all the sudden hill climbing due to RL? I'm not certain, and there is a reasonable argument that even if something in the train-set contains near-identical but not exact copies of test data, it's still generalization. I am sympathetic to that. But, I also wouldn't rule out that GRPO is amazing at sharpening memories along with math skills. At the very least, the above show that data decontamination is hard. Never ever underestimate the amount of stuff you can find online. Practically everything exists online.

I asked #R1 to visually explain to me the Pythagorean theorem. This was done in one shot with no errors in less than 30 seconds. Wrap it up, its over: #DeepSeek #R1