terminal bench · post-training & evals against global banality prev @ltiatcmu @columbia

sf
Joined January 2024
i’m at #COLM2026 all week 🩵 - grew up in seoul, did school in ny + pittsburgh, moved to sf 2 yrs ago - previously did post-training for knowledge work, now building coding rl envs - care deeply about data research (!) if you think about data selection / utility / heterogeneity / curricula or are synthetic data-pilled, i really want to meet you ☕️let’s chat over coffee / sanity walks around union sq / pilates / corepower, dm pls
2
1
48
2,995
1939 debate on formal consistency vs empirical reliability *cough* evals *cough* AI safety
I've always found this conversation between Wittgenstein and Turing (reprinted in Andrew Hodges's Turing biography) endlessly fascinating: "WITTGENSTEIN: … Think of the case of the Liar. It is very queer in a way that this should have puzzled anyone – much more extraordinary than you might think. … Because the thing works like this: if a man says ‘I am lying’ we say that it follows that he is not lying, from which it follows that he is lying and so on. Well, so what? You can go on like that until you are black in the face. Why not? It doesn’t matter. … it is just a useless language-game, and why should anybody be excited? TURING: What puzzles one is that one usually uses a contradiction as a criterion for having done something wrong. But in this case one cannot find anything done wrong. WITTGENSTEIN: Yes – and more: nothing has been done wrong. … where will the harm come? TURING: The real harm will not come in unless there is an application, in which a bridge may fall down or something of that sort. WITTGENSTEIN: … The question is: Why are people afraid of contradictions? It is easy to understand why they should be afraid of contradictions in orders, descriptions, etc., outside mathematics. The question is: Why should they be afraid of contradictions inside mathematics? Turing says, ‘Because something may go wrong with the application.’ But nothing need go wrong. And if something does go wrong – if the bridge breaks down – then your mistake was of the kind of using a wrong natural law. … TURING: You cannot be confident about applying your calculus until you know that there is no hidden contradiction in it. WITTGENSTEIN: There seems to me to be an enormous mistake there. … Suppose I convince Rhees of the paradox of the Liar, and he says, ‘I lie, therefore I do not lie, therefore I lie and I do not lie, therefore we have a contradiction, therefore 2 × 2 = 369.’ Well, we should not call this ‘multiplication’, that is all. … TURING: Although you do not know that the bridge will fall if there are no contradictions, yet it is almost certain that if there are contradictions it will go wrong somewhere. WITTGENSTEIN: But nothing has ever gone wrong that way yet. …"
8
711
so much alpha in picking one hard worthwhile problem and going deep
Introducing DeepRSI, a new industry standard for measuring recursive self-improvement. On preexisting benchmarks, tasks are contributor-based and short-horizon (~12 hrs). In DeepRSI, we push agents to work 48 hours on ML tasks actually derived from ongoing proprietary research.
13
1,535
data curation is the last remaining moat and i will die on this hill
7
1
55
2,762
democratizing end-to-end training!! congrats @KJHMiao @bryel_labs 🚀
We’re on a mission to build a billion AI models, not trained by us, but by you. Introducing @bryel_labs we help your company build its in-house AI lab and give you the software to make model development and continual improvement simpler. We are backed by @ycombinator and have trained models at Apple, Framer, and F500s. We’ve enabled companies to train: • Classification models that are 20x faster • Search models that cost 5x less. • Long horizon agent models that match the frontier at 5-8x lower cost. Are you interested in partnering with us? Reach out. bryel.ai/
6
674
cannot fathom how much painstaking iterations it must have taken to refine this pipeline, respect 🫡
How do you build the most diverse UMI dataset in robotics? In the last 8 weeks, we scaled an international UMI data collection operation from 5 to 90 operators, built out a dataset of over 1M+ unique tasks, and continue to capture over 45k+ new tasks each day. Robotic World Models require much higher diversity of tasks than imitation learning. Here's the why and how behind our operations. 🧵
1
14
1,023
Su Park retweeted
Today we're launching WebCloneBench: a benchmark which measures a model's ability to explore, copy, build and test various web apps. For each task, we give the model a reference site from one of Gmail, Google Sheets, Jira, DocuSign, Shopify Admin, Craigslist, Figma, Slack and Zendesk. It is instructed to build a fully feature-complete clone of every part of the application with an 8 hour time limit.
11
18
5
89
9,105
🧃refreshing take on visual tool use
Introducing Blender Bench v1 (BBv1): a benchmark for frontier LLMs on 20 realistic 3D production workflows including animation, modeling, rigging, cloth & surfacing. GPT-6 Astra leads overall; Claude Opus 5.5 wins modeling & cloth simulation.
7
810
and nearer and nearer
7
419
terminally benchmarking
1
23
956
accidentally said app instead of domain-specific harness and they kicked me out of SF
107
370
58
8,775
246,258
Su Park retweeted
not now honey, I'm doing frontier AI research
46
109
18
1,916
102,543
for 2026 F/W, hillclimbing endogenously constructed safety walls is in vogue
iteratively hillclimbing an artificial wall, v5 over v2 unequivocally suggests gains in long-horizon endurance
3
1,133
claude code's korean is bizarrely informal and uncouth. granted i rarely speak korean to it and have nothing in CLAUDE.md specifying register, but the gap vs english is wild curious how much of this is the korean training distribution vs how korean users interact with coding agents vs multilingual post-training just being bad at register
1
5
1,093
decided against posting the exhibits because the word choices are way too unrefined/nsfw. nuance matters. coding efficiency and multilingual register clearly don’t move in lockstep
648
quintessential sf experience – at San Francisco, CA
5
825
i regret to inform you that harness matters
With data, you can/should train both model weight & harness. 🐳: arxiv.org/abs/2609.00196 A full blown thread is on its way 🙂
9
1,412
good evals can be intentionally bad gradients + do not underestimate the power of thoughtful task curation
Demystifying Reinforcement Learning Post-Training of Language Models A very useful exploration of RLVR in a simplified setup, lots of good insights... "Result 1: sparse RL cannot find what the model never samples" "Result 2: dense rewards break that ceiling" "Result 3: "spurious rewards" are a story about the prompt set" "Post-training is best understood as redistributing probability mass inside the pretrained distribution. That framing suggests the measurement: track the probability the model assigns to the behavior you want, and track the entropy of its output distribution, throughout training." project page: minjang10.github.io/demystif… code: github.com/sankarh-1/demysti… link: arxiv.org/abs/2608.24949
6
1,464
back @ greed-ridden intellectual city – at San Francisco, CA
1
11
1,290