Banger paper introducing Jev-as-a-Judge.
The overall finding is that you want to use a cheap judge for most of your evals and send only the uncertain calls to a frontier model.
This paper measures how well that works with JEV, TypeSafe AI's decision-only judge.
On 510 held-out preference pairs, a cascade that accepted JEV's confident verdicts and escalated the rest to GPT-6 Astra kept 99% of GPT-6's accuracy at about 57% of its fee.
JEV returns a verdict and label probabilities with no reasoning text.
It costs $0.044 per 1,000 judgments at a median latency of 0.152 seconds, against $12.182 and 1.885 seconds for GPT-6, about 277 times cheaper.
On ordinary preference and evidence-grounded factuality it stays within 3 points of GPT-6 (92.2% against 93.5% on RewardBench, 87.5% against 86.7% on HaluEval).
The gap grows to 9 to 20 points on tasks that require checking a derivation or rejecting an elaborately written wrong answer, such as JudgeBench (78.6% against 93.1%).
On several benchmarks, JEV's gap to GPT-6 is concentrated in its low-confidence decisions, which is why the cascade works.
The escalation threshold did not transfer for every fallback model, so the authors recommend setting it on your own data.
Paper: academy.dair.ai/papers/jev-a…
Liam retweeted
This is the most valuable free resource we've created on AI Evals 🎉 (not exaggerating!)
I organized our public materials into this guide. It allows you to find answers to your eval problems w/o searching aimlessly.
Humans: pick the row that sounds like you.
Agents: point it at the post
The material draw on 60+ hours of office hours from our Evals course, where @sh_reya and I have taught 5k engineers and PMs Evals.
We add new material often, with 15 FAQs added in the last two weeks. Recent additions are marked with a "New" badge. You can find all of these and more here:
hamel.dev/blog/posts/evals-f…
Liam retweeted
Yesterday I was talking to the founder of a Series B AI-native CRM that’s raised $20M+.
They’re taking over the market right now, and he shared a super smart outbound playbook I had to share.
Step 1: Define the TAM pretty broadly.
Step 2: Figure out which CRM every company in the TAM is using.
Their trick?
Job postings.
Sales job descriptions constantly include things like:
“HubSpot experience required”
“Pipedrive knowledge is a plus”
“Salesforce proficiency preferred”
By scraping those job postings, they can map which CRM each company is using with surprisingly high accuracy.
Step 3: Tailor the outreach based on their current CRM.
If a company uses HubSpot, they send them a dedicated resource like:
“Why switching from HubSpot to [their CRM] is a no-brainer for your sales team.”
Same playbook for Salesforce, Pipedrive, etc.
Step 4: Cold call everyone.
And on the call, they mention the CRM the prospect is currently using.
It immediately shows they’ve done their homework.
Step 5: Conversion rates go through the roof compared to blindly blasting their TAM.
Simple, but very smart.
The real edge in their outbound isn’t the copy.
It’s the data they start with.
And you can basically automate this entire playbook with @GojiberryAI.
You can track companies based on the tech they use without scraping job posts, find the right people, and launch the outreach from there.
Hope you enjoyed this playbook!
If you think it could help another founder, feel free to share it 🙏
Liam retweeted
Today, we are launching IRIS, our open-source paper-finding agent.
Ask IRIS a literature question in plain English, with all of its conditions attached. It returns a ranked list of papers that were checked against every condition, one by one, with the sentences it relied on quoted next to each result.
Keyword search cannot do this. It returns papers that happen to use your words, misses the ones that say the same thing in other words, and has no way to check a condition like "evaluated on CASP targets after 2022," so you end up opening papers one at a time to check for yourself.
IRIS sends each question down the path built for it. Venue, year, author and citation conditions become set arithmetic over the corpus and are executed exactly. Topical questions get dozens of separate searches, one of which writes a hypothetical abstract of the paper that would answer the question and searches with that instead, and every candidate the searches return is scored against each condition on its own.
We ran IRIS on PaperFindingBench, the paper-finding task in AstaBench. The benchmark asks 66 literature questions of three kinds, each with a list of the papers that answer it, and scores a system on how many of the right papers come back and how highly they rank. A language model grades any returned paper the list does not cover.
IRIS scores 0.382 on the validation split at $0.13 a query, the strongest result published on that split. On questions naming one specific paper it scores 88.00, and on venue, year, author and citation questions it scores 68.24. Topical questions with several conditions attached come in at 22.78, and the reason is retrieval, since a paper that never enters the candidate pool cannot be ranked out of it. About 30 percent of the correct papers are never returned by any search built from the question's own words.
Everything we benchmarked is the code we published, under an MIT license, so you can clone IRIS and run that configuration yourself today.
Liam retweeted
For over a year, my team has been working with AI agents in a fundamentally different way than most.
All of our Claude/Codex/Pi/etc agents run in the cloud, and any session is joinable by anyone on the team.
Each of our meetings has an agent that launches subagents to do research, draft posts, and implement features as we discuss things.
A Chief of Staff agent lets me know what's waiting for my review, and can talk to any person or agent on the team to resolve bottlenecks.
Working in this fundamentally multiplayer way has given us a preview of the way everyone will work soon, so I wrote up a short manifesto explaining
• the current problems
• principles for a great solution
• some things that are tricky to get right
Check it out, and let me know what you think!
multiplayer-ai.com
Liam retweeted
We have several LangChain-led and community-led meetups coming up! RSVP and bring a friend!
🇸🇪 9/8 Stockholm
🇺🇸 9/9 Pittsburgh
🇺🇸 9/15 Chicago
🇵🇱 9/15 Warsaw
🇵🇱 9/22 Wrocław
🗽 9/22 NYC
🇫🇷 10/5 Paris (w/ @hwchase17)
🇳🇱 10/6 Amsterdam (w/ Harrison Chase)
🇩🇪 10/27 Munich
RSVP 👉 luma.com/langchain
Liam retweeted
I'm 30. I built an AI startup called @GojiberryAI doing over $4M ARR in one year. Got accepted into YC.
No one wants to hear this, but there's no magic distribution channel.
We got there by experimentmaxxing one channel at a time, based on our MRR level.
$0 → $6k MRR: pure outbound. Cold email + LinkedIn, using our own early product on ourselves. No brand, no audience — just us reaching out to people showing intent and starting conversations. Ugly, manual, effective.
$6k → $25k MRR: Reddit. Our first real acquisition breakthrough. We posted educational breakdowns in SaaS subreddits and pulled 10M+ organic views. Traffic quality wasn't amazing, but the volume flooded our trial funnel for months. Cost: basically zero.
$25k → $75k MRR: content + free blueprints. All-in on LinkedIn content, YouTube, motion-design videos, and giving away our internal systems as free "blueprints." Content got the reach, blueprints earned the trust, and a chunk of readers converted. This is where founder-led content started compounding.
$75k → $150k MRR: partnerships + X. We added Twitter, B2B influencers, sponsored newsletters, and a lifetime affiliate program that became a major lever. We also joined YC around here — the intensity went vertical.
$150k+ MRR: paid + hiring. Meta ads, Google ads, influencer agencies, and our first serious hires: growth, sales, engineering, product.
To get to $1M+/month, it’s mostly going to come down to hiring the right people and putting the right structure in place.
Here's the part people miss: none of these stages replaced the last one. They stacked. Outbound never stopped. Content never stopped. We just added the next lever once the current one was clearly working.
If you take one thing from this: don't chase five channels at once.
Beat one until it works. Then add the next.
This last wedge has been an awesome run!
I got a total of 4,552 OpenAI requests and 491,649 tokens used on a @LangChain_OSS niche multi turn intake agent that went into multiple real potential customers hands to test
A new personal record across the board 🦾
First Demo is back😀
The business operators who no showed yesterday simply had something unexpected come up.
Glad I added rules for following up as part of the early process! Been helpful for me so far on smol sample size
Seesh! My first demo no showed after two previous calls and a time they picked.
I built a demo instance custom to his business..
Brutal ☠️
Learning moment to improve my stack but not a kill shot.
Went back into trenches and found a new warm lead to refill the funnel.
I made a good decision today:
Share my agent with a friend before my first demo tomorrow.
My friend owns an "adjacent enough" workflow so I shared it and asked for feedback.
Demo improvement ✅
Buyer perspective sales insight ✅
More testing data ✅
Liam retweeted
LangChain Community Spotlight: Paper2Any 📄✨
Transforms research papers and PDFs into editable presentations, diagrams, and scientific figures using agent-based workflows. Open-source platform with multi-LLM support and production-ready deployment.
Check it out: github.com/OpenDCAI/Paper2An…
First demo booked🫶
I turned my failure into a success just minutes after.
How?
Immediately applied the lesson I just learned.
Followed up right then with the next hard earned positive discovery signal. The team remembered me and we booked a demo for this week.
I got hung up on by a warm prospect in my customer validation pipeline. Immediately. Twice💀
How did I go from a hard earned "I'm interested" after the first convo to hangups?
Lesson learned...
Follow up in between the physical validation funnel checkpoints so they stay warm
Customer 🔛Development🔒
For me, thinking about it like a system makes it much easier to get out of the comfort zone and execute because each iteration has an outcome that is completely independent of the entire system
Hacking around on an early business context system for my current LangGraph Agent 🧐
Deterministic FAQ extraction and answers against a config file with a fallback LLM call that builds a prompt with:
-human message (FAQ intent path)
-business context from config
Is this the right prod pattern for MVP?














