SEPT ’26: HUMANS STILL VERY MUCH HAVE JOBS. AI isn't even close to meeting the standard required of the most junior professionals in banking, advisory, and private equity today. ATLAS Finance agents must navigate complexity like a human does: "Johnny you just got staffed - check your email." Human: 100%. Best AI: 12%. Agents have access to data rooms, email, chat, calendars, docs, notes, Excel, etc. Our real finance professionals pushed the realism frontier further by building dozens of coworker and client personas that introduce ambiguity, (controlled) contradictions, and real-time updates that must be adjudicated to successfully complete the client-ready deliverable. We gave 11 frontier models 100 expert-level tasks that each take humans 15–30 hours. On Wall Street there's a saying, "If it's 95% right, it's 100% wrong." Autonomous knowledge work outside software still has a very long way to go.

Handshake AI CSO @Sbhaiwala03 reveals how they build simulated white-collar work environments to train frontier AI agents: "An environment consists of a few elements. The first is the software tools the agent needs access to. If I'm an investment banker, I have access to Excel, SEC EDGAR where all the financial reports are, and PowerPoint. Agents need the same access." "We build simulated versions because of infrastructure and access constraints. Labs are querying that tool thousands of times in a given second." "The second thing you need is data that populates the environment. If I'm building a financial model, I need the data room with the messy files from the company I'm evaluating." "The final piece is tasks and verifiers. What are the requests of the agent that would reflect what a real investment banker might ask. We try to find tasks complex and realistic enough that the models fail. If the model can't perform, we know it needs more training data." "What we send to labs is essentially a Docker file, a full containerized task. It has the tools the model has access to, the full data that populates the world, and the task and verifier. We'll send tens of thousands of these to a given lab for just one domain." @joinHandshake @GarrettLord

i made some updates to pavlov’s list and am pleased with them! thanks to @xeophon @ybenpan @phoebeyao @amit05prakash @kate_shapova @timshi_ai @kevinhou22 @madiator @dlbydq @daljeet_v @ingmariusX @davidstutz92 & Dylan Rogers for feedback lately (pavlov's = list of rl env cos)

We’re releasing Frontier-Bench: a benchmark that measures and evolves with the frontier of agent work. Built by the team behind Terminal-Bench and Harbor, Frontier-Bench is an on-going community effort. Frontier-Bench v0.1 contains 74 tasks on which the best agents score ~34%

AI models pose serious child-safety risks. While many model developers evaluate for explicit abuse material, other child-safety failures begin upstream: when a model helps an adult manipulate, impersonate, profile, or isolate a minor; or when it deepens a child’s emotional dependence on AI. Today we released CAREBench (Child AI Risk Evaluation), a new benchmark to assess such upstream child-safety risks in any language model. We provide: - 500 prompts spanning 12 risk categories (including grooming, relationship engineering, deception, extortion, AI anthropomorphization, and emotional dependency). - A model-response grader built from acceptability annotations by parents, clinicians (PsyD), and the Prevention Director at an accredited Children’s Advocacy Center. - Evaluations of 7 frontier models including Claude Fable, revealing failure rates ranging from 2% to 58%, with substantially different failure patterns across risk categories. This project exemplifies the type of vital work routinely performed by our AI Safety team at @joinHandshake

Grading agent rollouts in rubric-graded RL environments is itself a hard task. Prior approaches pass serialized artifacts or agent trajectories to an LLM judge; this loses information / doesn't support sophisticated criteria. In contrast, we built a reactive agentic judge.

Packed room to hear @alexgshaw and @ryan_marten break down how @harborframework grew into *the* framework for RL environments. In our RLEval workshop at @CAISconf today, attendees tackled big open challenges in RLEs & Agent Evals + I shared the approach we take at @joinHandshake