Take Control Of Your Data. Join our Data Science Briefing newsletter for the best in #DataScience and #MachineLearning
Manhattan, NY
Joined February 2015
- Tweets7.6K
- Following461
- Followers1.5K
- Likes351
Pinned Tweet
17 days until the LLM Eval Harness workshop.
Build a production eval harness live in 4 hours. Calibrated judges, error bars, CI gates.
Engineers who measure ship faster.
maven.com/bruno-goncalves/ev…
If you need a framework to make your AI agent "safe," your agent isn't safe.
Safety is engineered into the system. Not imported from a package.
Curious what others think — am I wrong?
(Building this live on Oct 31 → luma.com/qwhit9op)
Free lesson — Wed Oct 21, 12pm ET.
Meet the 3 primitives of Jev on a real ticket. One ~100ms call, $0.042/1M tokens, output free.
Register: maven.com/p/1d62f8/jev-in-on…
1/ Most AI agents fail in production not because the model is bad — but because nobody engineered the system around it.
Here's what actually breaks (and how to fix it):
🧵
3/ Problem 2: The context window fills up and the agent starts forgetting — silently.
Token budgets, memory tiers, and conversation compaction aren't optional. They're how you keep the agent from going off the rails 45 minutes into a session.
4/ Problem 3: There's no retry logic, no cost ceiling, no audit trail.
One malformed query. One infinite loop. One $400 API bill later — you wish you'd built run supervision from the start.
I'm teaching this on Oct 31. 4 hours, hands-on, Python only.
→ luma.com/qwhit9op
I'm running a live 4-hour workshop on Oct 31: Build an AI Data Analyst you can actually trust — MCP tools, guardrails, context management, fault tolerance. Laptop-only, framework-free Python. 🎃
Details + tickets: luma.com/qwhit9op
1/
Deeper on the one failure that actually ships:
fabricated numbers.
How an agent reports a 12% revenue drop that never exists, and the exact guardrail that stops it -
3/
Then it verifies:
- value matches the query output
- unit is right (not $12M reported as 12)
- reporting period is complete
LLM judging is reserved for clarity - not for 'is this number real.'
4/
The rest of the harness does the same job for the other failure modes:
permissioned tools, context budgets, injection defense, retries, traces.
It's a 4-hour hands-on build, framework-free, on a laptop. Sat Oct 31 - 19 days out:
Register: luma.com/qwhit9op
19 days.
Sat Oct 31 - 4 hours, live - we build an AI data analyst you can actually trust, in plain Python. No GPU, no cloud, no API key.
$149.99, seats are limited.
Register: luma.com/qwhit9op
Last call — free LLM eval gate session is TOMORROW (Oct 7, 2 PM EDT).
Regression thresholds + GitHub Actions wiring + merge blocking, all in 30 min.
Register: maven.com/p/07c4b3/stop-bad-…
Hot take: most of your AI agent's reliability problem isn't the model.
It's that you're letting the model pick its own tools and trust its own numbers.
The model should SUGGEST. A harness should DECIDE.
Am I wrong?
Hot take: if your LLM app doesn’t have a CI eval gate, you don’t actually know if it works.
You know if it worked last time someone manually checked. That’s not the same thing.
An AI agent isn't reliable because the model is good.
It's reliable because of the harness: permissioned tools, context budgets, guardrails, and recovery.
The model suggests. The harness decides.
Building that harness, framework-free, Oct 31:
luma.com/51oqj4cn
1/
Everyone's building AI agents.
Almost nobody's building the part that decides whether one is safe.
The model is a commodity now. The harness is the craft.
Four things a production agent needs that most tutorials skip -
4/
Guardrails + Recovery.
Separate data from instructions (prompt-injection defense), inspect every SQL statement, and when things break - retries, execution budgets, structured traces.
That's the whole harness. If you want to build it (framework-free, on a laptop), I'm running a 4-hr hands-on workshop Oct 31:
Register: luma.com/51oqj4cn