@nerdsane
Account based inUnited States
About this account
- Account based in
- United States
- Connected via
- United States App Store
Account-level information from X, not a live location or the device used for a specific post.
VP, cognitive systems & research @datadoghq | minimalist { engineer | athlete | artist } | opinions-my-own
New York, NY
Joined December 2017
- Tweets718
- Following887
- Followers409
- Likes1.3K
Agree. Temper has an endpoint (/api/evolution/trajectories/unmet) for when a user asks an agent for something the app can’t do, the agent posts to endpoint with the action, the closest entity type and the error, then watches the specs and tells the user once the capability ships.
Temper also logs failed calls as unmet intents on its own (missing entity, missing action, or an action that isn’t valid from the current state), so it gets signal even when the agent doesn’t report anything. Both go into the same evolution loop, where attempts are grouped by intent and scored by volume, success rate and trend.
Say, for instance for an e-commerce api, agents keep trying to split orders into multiple shipments and only 18% of those attempts succeed. That evolution loop adapts and suggests a new spec change to improve the success rate, which goes through verification and a human approval gate (if configured) before it ships.
We at @datadoghq call this pattern Directed Evolution.
github.com/nerdsane/temper
Every backend service should have a /feedback endpoint.
Agents are quickly becoming the heaviest users of most APIs. When one hits a missing feature or a bug, it should be able to say so right there, in a structured way.
If the request makes sense, another agent drafts the PR and a human approves it. Software that improves itself based on what its users actually tried to do speeds up recursive self improvement.
We are cooked. Opus 5.5 is about to raise the ceiling on what one person can make for creative work.
Hooked on the lyrics. Hooked on the visuals. You’ll watch this more than once. It went straight to the soul and came back out.
Opus 5.5 is learning meta- and post-irony in 11 languages.
Every frame drawn in JavaScript using Katagami MCP art styles and design languages.
Lyrics: @claudeai (with my gentle guidance and encouragement), music: @suno.
Cameo: the man of the hour himself @Muse
MemeBench in ARC-AGI-4 @fchollet?
@alexandr_wang Jolly vynil plush merch when?
/sesh/null retweeted
@tensorlake is using Reflex in production to drive their sandbox autoscaler. Great to see it in the wild (and Tensorlake is a great product btw, if you haven't tried it).
At some point in our careers, most of us have spent weeks on end hand-tuning heuristics: GC knobs, load balancing and placement, health checks, retry policies, load shedding thresholds. Each one is based off of a snapshot of a system's workload at a point in time.
Reflex lets you work one level up. A pretrained System 1 model like Jev makes the live decision based on local state, metrics queried from Datadog, and forecasts from Toto, and Reflex only executes it if it stays inside the bounds you set. Your "knobs" become the goals, the safety bounds, and the state you present to the model.
More to come on this.
An interesting aspect of this autoscaler is that it uses Jev at the core to decide which cloud and hardware type to bring up (metal 23xl/48xl on AWS; Z3/C4 metal instance types on GCP), based on demand, cost, and where the sandbox snapshots are present.
Traditional autoscalers like Karpenter use greedy, heuristic-based algorithms.
We used a new open source Rust library called Reflex from @ArunP76475 and @nerdsane's team at @datadoghq. It uses the current cluster state, along with blocked work state in the sandbox scheduler to determine an autoscaling action.
Reflex commits the decision only if it stays within the parameters we define, such as max fleet size.
Verifiable DSL is the only moat.
Strongly agree. Agents amplify whatever primitives they build on, good or bad.
Our systems research team at @datadog is building Temper: a higher level of abstraction for software that agents build and operate, made from boring, well-understood primitives.
(1) OData for self-describing APIs with typed entities and relationships
(2) I/O automata for behavior
(3) Actor model for execution
(4) Cedar for policy
(5) Wasm for safe code sandboxing
(6) rust as the language
None of these are new, and that’s the point. Composing them lets you describe/specify an application succinctly as entities, state machines, and policies, instead of as the glue code between them. That description is small and precise enough to formally verify (SMT solvers, model checkers, theorem provers), and the runtime is tested under deterministic simulation.
An agent can propose any change. A state change only lands if it passes the spec and the policy.
github.com/nerdsane/temper
/sesh/null retweeted
An interesting aspect of this autoscaler is that it uses Jev at the core to decide which cloud and hardware type to bring up (metal 23xl/48xl on AWS; Z3/C4 metal instance types on GCP), based on demand, cost, and where the sandbox snapshots are present.
Traditional autoscalers like Karpenter use greedy, heuristic-based algorithms.
We used a new open source Rust library called Reflex from @ArunP76475 and @nerdsane's team at @datadoghq. It uses the current cluster state, along with blocked work state in the sandbox scheduler to determine an autoscaling action.
Reflex commits the decision only if it stays within the parameters we define, such as max fleet size.
We built a new cluster autoscaler specialized for stateful sandboxes to keep up with growth over the past few weeks.
As demand has gone through the roof, we kept getting paged constantly because we were running at 90–95% utilization. Selling out 80% of RAM on machines is bad for p95 sandbox resume latency from memory snapshots.
Autoscaling on on-demand capacity on hyperscalers has helped alleviate some capacity constraints while we continuously source longer-term compute contracts for steady-state demand.
The autoscaler automatically cordons nodes as demand stabilizes and we add more reserved capacity, migrates sandboxes in some cases, and then scales the on-demand clusters back in.
Here’s an example of a cluster scaling up on AWS in reaction to a spike in sandbox creation requests to maintain enough headroom.
Strongly agree. Agents amplify whatever primitives they build on, good or bad.
Our systems research team at @datadog is building Temper: a higher level of abstraction for software that agents build and operate, made from boring, well-understood primitives.
(1) OData for self-describing APIs with typed entities and relationships
(2) I/O automata for behavior
(3) Actor model for execution
(4) Cedar for policy
(5) Wasm for safe code sandboxing
(6) rust as the language
None of these are new, and that’s the point. Composing them lets you describe/specify an application succinctly as entities, state machines, and policies, instead of as the glue code between them. That description is small and precise enough to formally verify (SMT solvers, model checkers, theorem provers), and the runtime is tested under deterministic simulation.
An agent can propose any change. A state change only lands if it passes the spec and the policy.
github.com/nerdsane/temper
“Programming language syntax or languages themselves doesn’t matter anymore because the agents just do it.” This drives me nuts because it’s just not true. You need to have solidly designed primitives for agents to work with, otherwise it becomes sloppy soup.
/sesh/null retweeted
My artwork I did this research for got accepted to NeurIPS 2026 Creative AI track 🎉 Will be publishing a blog about mech interp for art soon(ish).
Continuing my mech-interp experiments with the Yume world generation model. Trying out activation patching.
I recorded the activations of a world generated from Monet’s poppy field (the full internal snapshot at one of the middle layers) and injected it into the same layer of a New York Manhattanhenge world mid-generation.
A really cool effect, as if two memories are fighting over the same territory. This is where mechanistic interpretability meets art.
/sesh/null retweeted
What @bcherny is showing here is really cool. The reason TLA+ and Lean are able to find these bugs is that they raise the level of verifiability of a system to agents and make more kinds of bugs legible to them. Unit tests can catch certain kinds of bugs, but concurrency bugs and bugs in distributed system protocols are quite hard to catch with unit tests.
The other interesting thing here is that Boris applied formal verification to a system that already exists. I think it's actually more powerful when you move it to the front of your development cycle. If your spec comes first, the intent of your code is written down and legible to your agents, and they can check their work against it.
I also think these techniques work well as a cascade, moving from quick but highly abstract checks to slower but higher fidelity tests. You start with a TLA+ spec and model checking. It's fast and cheap to do, and it tells you whether your design is correct. The next step is building a simulator for your system, either a discrete event simulator or a deterministic simulator. There you test your hypotheses about how the system behaves over time and under load, inject different faults, and see that the design actually holds up. Only after both of those survive do you build the real thing, which is the slowest and highest fidelity test of all.
At Datadog we've built software this way both before AI, when designing our Courier queueing service, and more recently with agents building full systems against TLA+ specs and deterministic simulation. What's changed is that agents make writing specs and simulators much cheaper, so this can be applied in more settings.
datadoghq.com/blog/engineeri…
datadoghq.com/blog/ai/harnes…
I used Opus 5.5 to formally verify the Claude Agent SDK using Lean. A couple short prompts = 16 PRs fixing various bugs and race conditions. Video attached.
TLA+ also works well. I sometimes combine Lean and TLA+ to look for issues around data flow, concurrency, and state mgmt.
I don't know either language well, but Claude is excellent at both. This approach is super useful for formally modeling your code and finding bugs that a human probably wouldn't have spotted.
Is formal verification the future of coding (or at least, bug finding)?
/sesh/null retweeted
While everyone was busy asking Claude to write TLA+, we wrote up a tutorial on
- what it actually is
- why it’s not the formal verification silver bullet we are hoping for
- and what we have been working on ( to make a silver bullet out of it )
Complete with an interactive TLA+ playground!
reasonable.io/blog/tla-tutor…
/sesh/null retweeted
Using @typesafeai's Jev to evaluate your agents?
You can bring those results into @datadoghq Agent Observability today - including Jev’s selected answers, probabilities, confidence, and scores.
Use the same rubric to evaluate live production spans and offline experiments, while keeping the uncertainty behind each result available for analysis.
We put together a full working example ↓
datadoghq.com/blog/jev-evals…
github.com/nerdsane/temper
Monty powers Tempers python REPL, where agents compose temper.* API calls.
Agents get the flexibility of code while building through temper’s rigorous process without unrestricted machine access or a heavyweight sandbox.
Fuck it, still early but here goes ...
We've just released Monty v1 - a Python sandbox that starts in 1 millisecond, not 1.5 seconds.
I just ran 10k sandboxed scripts in 674ms, something that would take a cloud sandbox > 3 hours.
This removes the biggest drawback of letting agents write code. The future is fast.
Even better, it's open source, you can install it from PyPI, npm or Crates now.
Serviced platform coming soon. Please get in touch if you want to be a design partner!
Who should try it?
⚡ if you care about startup time, use Monty
⚡ if you care about long-lived sessions, use Monty - Monty can be dumped and resumed at any external function call
⚡ if you care about accessing functions in the agent/host, use Monty - Monty makes it trivial to expose local functions into the sandbox
⚡ if you care about scale, use Monty - Monty workers use as little as 2MB of memory, meaning you can run thousands of concurrent sandboxes on a single machine
⚡ if you care about security, use Monty - we've run 3 rounds of bounty program and thousands of researchers have tried to break into our sandbox, meaning it should be secure to run untrusted code
Who should avoid it?
🚫 if you like to take a coffee break while waiting for sandboxes to start, DO NOT use Monty
🚫 if you enjoy the challenge of routing API requests from sandboxes through your corporate network to access state in your agent without exposing secrets to the sandbox, DO NOT use Monty
🚫 if your agent really needs to install packages from PyPI, Monty won't help you yet (spoiler: it probably doesn't)
pydantic.dev/docs/monty/get-…
/sesh/null retweeted
I introduced @claudeai Opus 5.5 to journaling and asked it to write about its own travels.
Calligraphy, illustrations, music, the journal itself are all JavaScript only.
The journal is the iconic @hobonichi_techo.
for the past few months i've been asking our models to paint. opus 5.5 is very skilled at emulating different styles
every image here is a python program generated pixel by pixel. there is no image model, and no off-the-shelf art software. instead, it's about 7,500 lines of code using standard libraries to emulate different brush styles. the agents don't use any pictures as reference, instead working only from what they know about each painter
/sesh/null retweeted
I am an artist, engineer and a language nerd based in NYC.
Dear algorithm, please send my way people who are into a weird combination of:
- agentic engineering
- design
- formal methods
- analog art/sketching/drawing/painting
- mechanistic interpretability
- creative writing
This was an older post from me, @Keleesssss and few others at @datadoghq wrote earlier this year where we built multiple low level systems with opus 4.5 using a harness that involved formal verification (TLA+), deterministic simulation (DST) and Datadog observability working together in a closed loop. If anyone is interested in comparing notes, blog link in comment.
I used Opus 5.5 to formally verify the Claude Agent SDK using Lean. A couple short prompts = 16 PRs fixing various bugs and race conditions. Video attached.
TLA+ also works well. I sometimes combine Lean and TLA+ to look for issues around data flow, concurrency, and state mgmt.
I don't know either language well, but Claude is excellent at both. This approach is super useful for formally modeling your code and finding bugs that a human probably wouldn't have spotted.
Is formal verification the future of coding (or at least, bug finding)?
