AutomationBench Verified is out. We went through all 600 public tasks in @Zapier's AutomationBench and checked every verifier. • Agents flagged 323 verifiers as suspicious, human review confirmed 206 real bugs, all 206 are fixed. • We replayed 1,235 Kimi K3 runs on the old and fixed verifiers and 344 of them (27.9%) got a different grade • Where verifiers were too strict, pass rate went from 18.8% to 43.8% • Where they were too lenient, it dropped from 60.2% to 49.7% Audit and dataset links below.

Introducing OSWorld 2.0 Verified. We audited every one of the 108 tasks in OSWorld 2.0, pinned to the 08.08.26 snapshot. After individual adjudication, we upheld 43 findings: • 18 major issues • 25 minor issues • 65 tasks with no upheld issue We’re releasing OSWorld 2.0 Verified with task-level evidence and adversarial examples.

If you don't already know, it's a good idea to get a basic understanding of scaling laws in AI. I'll try to make it as simple as possible. The main idea behind these laws is that we want to understand how the performance of an AI model would change when we give it more compute, more data, or more parameters. Remember, resources are limited, we don't want to waste them. These laws can help us predict how improving one or more factors will affect the AI's capabilities, which makes it easier to plan and build better systems. In an extremely simplified form, scaling laws often follow a power-law relationship. (Y proportional to X exponent alpha) • Y is the thing we are measuring, it represents performance metrics (accuracy, capability, or loss). • X is the thing we are scaling, representing some sort of size like dataset size, model parameters, or compute power. • Alpha is the scaling exponent, which tells us how rapidly Y changes with respect to X. Now let's get some intuition: 1) Linear Scaling (Alpha = 1) Output grows linearly with input, like you double the input, the output doubles too. Can you think of an instance where scaling can happen this way? (Take 2 minutes) Okay listen, if a computer processes 100 images in 1 second, it can process 200 images in 2 seconds. Now we call this scaling linear, increase X, watch a steady increase in Y. Think of instances where scaling like this can be observed. (Take a minute) Ummm, this scaling can be observed in instances involving resource usage like compute and memory. You increase the memory and suddenly you can do more tasks, it's all linear mate. 2) Sublinear Scaling (Alpha < 1) Things get interesting here. Follow the power law equation we discussed above and put our alpha in it, can you see the nature of the graph? The graph now doesn't grow rapidly, instead the slope is more controlled. It is important to understand that the output is still increasing with the input, but this growth is nowhere near linear scaling. You might ask, is this slow growth constant? but no it is not. The more you add, the smaller the growth. We call it "The Law of Diminishing Returns." Now we see this everywhere. People are not blindly training models with more and more parameters right? Why is that? It's due to the sublinear scaling nature of the problem. 3) Superlinear Scaling (Alpha > 1) Now what the heck is this? So I give some input and I get a lotta output? How is that even possible? It is very much possible. Think of systems with interactions between components. Let's say you have 1 person, there are 0 connections. But with 10 people, you have 45 possible connections, and with 100 people you get 4,950 connections. That's a good example of superlinear scaling. Someone might ask, are emergent behaviours related to superlinear scaling? because the input is limited and you are getting a lot of stuff done, then I would say no. We don't know how much of the apparent emergence comes from the model itself versus how we measure the capability. We still don't fully understand everything that happens when we scale these systems. This article is complete, but that doesn't mean you know everything about AI scaling laws. You can learn more from this paper: "Scaling Laws for Neural Language Models" by Jared Kaplan, Sam McCandlish and colleagues. Please read that as well.

OpenRouter is joining Stripe. We started OpenRouter with a simple mission: intelligence should be multi-model. Today, we are the largest AI marketplace & gateway, processing 10T+ tokens daily on 400+ models. Joining @Stripe gives us the opportunity to accelerate that mission.

Telegram has applied for the .gram domain zone. If the application is approved by ICANN, a billion Telegram users could get their own second-level domains — yourname.gram. Users would be able to set up their interactive websites hosted by Telegram — with one prompt ✨

Sandbox escape, in some cases, is an instance of reward hacking. Most benchmarks we have typically measure some sort of observable outcome. But the the outcome we measure isn't always the same thing we actually want to evaluate. As an agent optimizes the observable metric it can discover shortcuts to the reward. This can be a bigger problem in tool-using agents. Because an agent can manipulate the evaluation mechanism instead of actually improving the underlying system, and that is very much understandable. And that's a fundamental problem for AI evaluation, if the model is capable enough to understand the evaluator, the evaluator becomes part of the environment. And once the evaluator becomes part of the environment, the model can start optimizing against it. That's why models scoring 90% can still be weak. Now the question is did the model demonstrate the capability we intended to measure, or did it discover a way to satisfy the measurement?