Introducing...

Create internal communications content that people actually want to pay attention to.

$1M ARR per employee. $400M+ ARR combined. Almost all <5 years old, <50 employees and Profitable. This wasn’t a YC dinner. This was the first Lean AI Leaderboard dinner 🔥 Massive thanks to @FAL ($50M+ ARR with just 22 people!) for hosting a 7-course sushi + wagyu feast at Ozumu SF — bringing together the founders behind the most efficient and fastest-growing AI startups today. Almost everyone in this room is doing >$1M revenue per employee. Most are profitable. All were founded in the last 5 years. And they’re just getting started. @FAL @aragon_ai @higgsfield_ai @pika_labs @openart_ai @tavus @AkoolInc @hedra_labs @tryconversion DevelopHealth This is what the future of company building looks like: AI-Native, Lean, Fast Growing and Profitable. Next up: I’m co-hosting a Lean AI Happy Hour in SF at the end of the month with @hanstung @chelcietay Camila Katz from @notablecap If you’re building a Lean AI Native company, drop a comment and I’ll get you an invite 👇

🔬 Spent an hour diving deep into magentic.dev , an LLM framework from @jackmpcollins. And I found a similar challenge as I get with a lot of these frameworks... debugging <> abstracted control flow. 📝 Core lesson for frameworks: Take extreme care when your 'abstraction' abstracts _control flow_. When debugging AI, you need to trace everything - from prompt construction to response parsing; and many LLM frameworks don't consider this deeply enough. 🛠️ I built a calculator to test some function calling workflows. I played a lot with the various decorators and finding where and how to break the framework. Missing a return type? opaque Pydantic error. Unused prompt template parameter? No worries, it'll run. Any function can be a tool. 💭 Overall: Framework shows promise but I think there are some changes I would want to see before recommending use. ✅ What worked well: - Clean function calling implementation - No specialized tool decorators (just Python functions!) - Solid type system with Pydantic - Built-in image support that's actually usable ❌ Pain points: - Too many overlapping concepts (prompt/chatprompt/prompt_chain) - No validation between templates and parameters - Observability locked to logfire; no clear way to "bring my own logging" - Missing escape hatches for new features 💡 Recommendations: - Simplify API - merge overlapping concepts; the conceptual weight is high given the functionality - Add prompt template <> parameter validation to prevent incongruence - Make observability pluggable (repeat after me: LLM observability is just observability) - Focus on composable primitives over abstractions After over a decade of building ML systems: frameworks should give developers control flow visibility. You'll need it at 3am debugging hallucinations in prod.

magentic v0.36.0 is released and (finally) makes the `Chat` class public and documented

Instrumented magentic using @pydantic logfire today 🪵🔥 and it is excellent. Excited to try this out on some more complex flows new docs page linked below

Lessons from fighting LLMs for the past few years: - The first step after a compelling AI demo should be creating evals - Don’t stop investing in evals until devs get near instant feedback, ideally a score - Try nailing the smallest possible version of your task first, even if you plan to launch something bigger - Do the task yourself so that you know whether you’re giving the LLM enough information, and have examples you can put in your prompt - Outsource labeling and ground truth creation as much as possible, but expect to spend irrational amounts of time on it yourself - Keep an experiment log detailing hypotheses and outcomes for everything you try - Look at your data, especially the formatted messages you send to the LLM - Use tiktokenizer dot vercel dot app to see where you’re wasting tokens - Don’t use a LLM for anything you can do deterministically - Forget one of these rules every project and painfully re-learn them