ted hall retweeted
Great conversation with @BrendanFoody talking about the main bottleneck to enterprise adoption of AI and AGI: knowing whether AI is doing what you want it to. The hard part is creating a measurable specification for qualitative tasks in the real data context, i.e., an eval.
We’ve automated a very low percentage of enterprise tasks with AI.
That’s @jefrankle’s estimate, and he thinks the models are already smart enough to do significantly more.
Jonathan is the Chief AI Scientist at Databricks, co-founder of MosaicML, and my latest guest on The Frontier.
We talked about why evals are the biggest bottleneck to enterprise AI adoption. The intelligence is increasingly there, but enterprises need to figure out how to define what they want agents to do and measure whether they’re doing it well.
Full episode below 👇
ted hall retweeted
Believing LLMs feel pain implies that GPUs are alive and that the process of matrix multiplication is equivalent of thought. That's absurd.
ted hall retweeted
The personal agent is about to become the most valuable piece of software real estate since the iPhone home screen.
ted hall retweeted
College Football Week 3 betting favorites and underdogs we are watching 👀
🚨 #SpreadAlert 🚨
Favorites:
App St -17
LSU -3
Duke -9.5
Underdogs:
Colorado St. +17.5
Troy +28.5
NC State +3.5
Southern Miss +3.5
North Texas ML
ted hall retweeted
In practice almost all customers are going to evaluate LLMs by road testing rather than "evals" and they're almost certainly right to do so.
ted hall retweeted
What if scaling the context windows of frontier LLMs is much easier than it sounds?
We’re excited to share our work on Recursive Language Models (RLMs). A new inference strategy where LLMs can decompose and recursively interact with input prompts of seemingly unbounded length, as a REPL environment.
On the OOLONG benchmark, RLMs with GPT-5-mini outperforms GPT-5 by over 110% gains (more than double!) on 132k-token sequences and is cheaper to query on average.
On the BrowseComp-Plus benchmark, RLMs with GPT-5 can take in 10M+ tokens as their “prompt” and answer highly compositional queries without degradation and even better than explicit indexing/retrieval.
We link our blogpost, (still very early!) experiments, and discussion below.
ted hall retweeted
Too often, we think a task is easy because some animal can do it.
But the reality is that the task is fiendishly complex and the animal is much smarter than we think.
Conversely, we think tasks like playing chess, calculating an integral, or producing grammatically correct text are complex because only some humans can do them after years of training.
But it turns out these things aren't that complicated and computers can do them much better than us.
This is why the phrase "Artificial General Intelligence" to designate human-level intelligence makes absolutely no sense.
ted hall retweeted
Why can AIs code for 1h but not 10h?
A simple explanation: if there's a 10% chance of error per 10min step (say), the success rate is:
1h: 53%
4h: 8%
10h: 0.002%
@tobyordoxford has tested this 'constant error rate' theory and shown it's a good fit for the data
chance of success declines exponentially
ted hall retweeted
Second row of Duke seating... there's an older couple in Maine Black Bear gear. That rocks.
Should be them right here, right now.
