Staff Engineer @ Braintrust. All posts are my own. Formerly at Zapier (Sr. Engr), omcode (founder), Code RGV (founder), Google, UCLA (M.S. EE), UTSA (B.S. EE)
USA
Joined June 2008
- Tweets2.3K
- Following673
- Followers546
- Likes806
There's still time!
Come by to The Midway to attend:
Builder Odyssey: AI Infra, Multimodal & FDE | Founder Demos & AI Film Showcase · Luma luma.com/seamate-tey7
I'll be sharing insights and lessons learned by building products for enterprise ai builders.
Olmo Maldonado retweeted
Your tokens have plans this Sunday 🤖
We’re bringing Builder Odyssey to SF on Oct 18—run, build, deploy, repeat.
@H0wie_Xu @jfischoff @ibolmo @jove @zylberstein
Grab your spot on @partiful 👇
partiful.com/e/0PAdTfFcp3hgB…
PSA: If a random security/permission prompts you to accept/deny, and you've got agents working. You *must* count to 10, and then check on the rascals.
Days of auto accept (because you're the one tripping the detection) are long gone.
Olmo Maldonado retweeted
GPT-6.1 Sol promises similar performance to last week's GPT-6 release at one-fifth the token cost of Astra. Does it deliver?
We re-ran our eval on how well models can guide users through multi-step problems. GPT-6.1 Sol reached 78.6% problem-solving accuracy at token prices far cheaper than Astra, Opus 5.5, or Fable 5.1.
The most notable improvement was how the new model guided users through teaching interactions. Our "teaching quality metric" for GPT-6.1 Sol rose from 64.1% to 77.6%, compared to Astra's 82.2%.
The results indicate GPT-6.1 Sol is able to interact with users in the ways they specify, leading to better pedagogy performance.
Read more → braintrustdata.link/gpt-6-1-…
New tactic:
One agent that only adds test (red team) to prove something is working or not working. Not allowed to edit non test code.
Separate agent that only changes code (blue team) but is not allowed to edit tests.
You're part of the red team.
Friendly reminder:
Send Braintrust all of your data (text or binary) with zero/zip/nada of setup (no schemas).
Brainstore will take it from there.
You (or your agents) can retrieve with (un)structured calls virtually immediately. Now even faster.
We built Brainstore for the amount of data agents generate. Now agents are helping teams search through traces and run far more queries than a person would.
Nitro is Brainstore's new asynchronous query execution engine. It separates waiting for storage from processing the data it returns, bypassing the limits of traditional object storage.
We ran Nitro on more than 1 million real queries and found that full-text searches were over 2x faster. It's already on for Braintrust SaaS and BYOC customers.
Read more → braintrustdata.link/nitro-bl…
Olmo Maldonado retweeted
We built Brainstore for the amount of data agents generate. Now agents are helping teams search through traces and run far more queries than a person would.
Nitro is Brainstore's new asynchronous query execution engine. It separates waiting for storage from processing the data it returns, bypassing the limits of traditional object storage.
We ran Nitro on more than 1 million real queries and found that full-text searches were over 2x faster. It's already on for Braintrust SaaS and BYOC customers.
Read more → braintrustdata.link/nitro-bl…
Olmo Maldonado retweeted
We tested where Jev holds up as a judge.
Jev was fast, cheap, and highly competitive for judging groundedness. But it lagged behind models with reasoning for math and code domains.
Use it for certain eval tasks, but don't throw out your LLM-as-a-judge just yet.
Read more → braintrustdata.link/jev-vs-g…
Jev from @typesafeai can replace your LLM-as-a-judge for scoring agent responses.
It returns a choice or numeric result with information about uncertainty, so you don't need to spend time and resources prompting a general-purpose model into an LLM judge.
Use Jev as a judge scorer in Braintrust and review its selected answer, confidence, and probabilities alongside the score. Trace Jev calls from your own application with the JavaScript or Python SDK.
Read more → braintrustdata.link/jev-eval
Olmo Maldonado retweeted
Jev from @typesafeai is now available as an evaluation model in Braintrust.
If you have an existing scorer, just change the model in the dropdown to Jev and reduce your scoring cost by 400x.
We're very excited about this and are doing more of our own evals in the coming days.
Jev from @typesafeai can replace your LLM-as-a-judge for scoring agent responses.
It returns a choice or numeric result with information about uncertainty, so you don't need to spend time and resources prompting a general-purpose model into an LLM judge.
Use Jev as a judge scorer in Braintrust and review its selected answer, confidence, and probabilities alongside the score. Trace Jev calls from your own application with the JavaScript or Python SDK.
Read more → braintrustdata.link/jev-eval
I've heard agents aren't good at product design. Try this:
Assign a few agents to create criteria for X.
Have a panel of agents score against said criteria.
I'm expecting a score of 40%. Adjust criteria until the panel agrees with my score.
Improve X until the score is 80%+.
Need to token max? Use "don't stop until".
"Check the CI, fix anything we caused, and don't stop until the CI is green"
"Use the feature specs and exercise them with Playwright MCP. Fix any findings. Don't stop until you've gone through all of the specs."
YMMV.
Back in my day conflicts were avoided like the plague.
Now with AI, I'm expecting Github to auto-fix conflicts with a suggested commit. For mavericks, maybe auto-commit the conflict fixes.
I've been testing letting the AI a "all" PR and for it to manage splitting up the PR and pushing changes done to the "all" PR down to each downstream child PR.
A couple of lessons learned:
4. Human reviewers lack the complete picture. Include links back to the design docs, and help orient them what is the portion of this Manhattan project that you're working on. They're seeing the tree from the forest on purpose, but it is still jarring.