Head of Product, LangSmith at @LangChain 🚀 | prev: @Harvard MS/MBA, @RocksetCloud (acq. OpenAI), @BainCapVC, @ContraryCapital
San Francisco, CA
Joined September 2017
- Tweets539
- Following639
- Followers1.2K
- Likes5.3K
Sam Crowder retweeted
some great new LangSmith docs on traces vs threads vs trajectories (new concept)
observability data is no longer just for observability - its also for memory & learning
having a really clear mental model of this data is incredibly helpful!
docs.langchain.com/langsmith…
Sam Crowder retweeted
the most common FUD our competitors throw against LangSmith is that it only works if you use our open source.
but actually as of this month, a MAJORITY of our active self-serve customers build without a LangChain open source framework / harness.
LangSmith is model, framework, and cloud neutral, and we have the numbers to show it!
Sam Crowder retweeted
Trace AI SDK 7 operations with LangSmith!
Sam Crowder retweeted
Trace data is literally worth its weight in gold these days, if you know what to do with it! As has been established, creating effective agents requires shipping early, observing behavior, and iterating quickly. At the core of this are your agent traces capturing exact inputs, outputs, steps, and metadata along the way.
Analyzing traces helps surface inefficiencies and areas for improvement, but they can also be used in more sophisticated ways to set up robust evaluations.
Here's two of the ways we use traces to build evals for production agents 👇
Sam Crowder retweeted
everyone is talking about self-optimizing loops in software & agents. but what does that actually mean?
in my mind, it's a system that observes it's own outputs, evaluates them, and uses that signal to improve itself in the future. the reason why it has become so popular now, is because the evaluation step is finally reliable with llms. this wasn't really the case a year ago.
this is why i'm so bullish on langsmith engine. we've incorporated a ton of different concepts that allow developers to invest in this self-optimizing loop that makes the improvement flywheel spin faster & faster.
some examples of this include
> feedback you leave on traces are automatically triaged
> every fix that we suggest has an online evaluator, so you never regress
> we create offline evals that you can add to your test suite
> we continually learn on your preferences, and tune our evaluation & fix step based on this
and we're seeing crazy adoption, and lots of growth across our customers. it is truly something that just gets better the more time you spend on it.
Sam Crowder retweeted
one of the best decisions i made was to start tracing my claude code sessions into LangSmith.
Has been a game changer to be able to share my conversations, track usage patterns, monitor cost.
And w new messages view its all suuuuppper easy to parse. def recommend trying it
Sam Crowder retweeted
We’re hiring for Labs! 🧪
If you’re interested in working with us to push forward Continual Learning, pls DM me with a blurb + link to the best Applied Research you’ve done (or even better shipped!)
you’d be a good fit if you have some previous research background and are excited to build real experiments on:
- understanding massive amounts of Agent Trace data
- building + updating Environments over time
- Harness Eng + Post-training over long time horizons
Sam Crowder retweeted
Agent observability is a means to an end: making your agent better.
But observability and evals tools have traditionally failed to connect traces to meaningful actions. Agent engineering teams are left combing through traces, guessing at root causes, and writing evals manually.
We built Engine to close the agent improvement loop. Engine monitors your agent's traces, creates ready-to-merge fixes, and writes evals.
Now every trace becomes a fix, an eval, and a better agent.
Sam Crowder retweeted
🌉Join us in SF for a meetup on building better agents with the agent and code improvement loops. You'll hear talks by @samecrowder, Head of Product at @LangChain, and @nnennahacks, AI Developer Relations Lead at @QodoAI .
Sam will walk through the agent improvement loop and how teams use traces as the foundation for continuous improvement.
Nnenna will present how Qodo is building agents, their architecture, and how they’re enhancing them using LangSmith.
🗓️ Wed, April 29 | 🕕️ 6 PM | 📍 SF (SOMA)
RSVP 👉️ luma.com/4nu6vpsh
Sam Crowder retweeted
When you're the system of record for 6.7B agent runs, you start to see things 👀
Recent observation - Azure's share of OpenAI traffic nearly 4x'd in under 3 months. The enterprise wave is arriving fast
The LangSmith Signal: Azure's share of OpenAI traffic grew nearly 4x in under 3 months.
We're sharing how devs are building agents, by the numbers. While most orgs started by connecting directly to OpenAI, over the past 10 weeks we've watched Azure's share of that traffic grow from 8% to 29%.
We've analyzed this trend via LangSmith Observability data across more than 6.7 billion agent runs.
Our hypothesis:
💡 Early adopters moved fast and went direct, but the enterprise wave is now arriving in force
💡 Azure gives teams the compliance, security, and procurement infrastructure they already have in place
💡 Azure traffic 4x-ing in 10 weeks likely indicates AI development is maturing quickly
Sam Crowder retweeted
Great stat in here: Claude Code went from 17% to 92% on our eval set once it had access to LangSmith traces and Skills. A coding agent without trace data is just guessing at fixes
New conceptual guide: 🔄 The agent improvement loop starts with a trace
Tracing is the foundational primitive for improving agents.
A trace gives you the full behavioral record of what an agent actually did. From there, teams can enrich traces with evals and human feedback, turn recurring failures into test cases, validate fixes before shipping, and repeat.
This guide breaks down the full improvement loop and why reliable agents are built through trace-centered iteration, not one-off debugging.
Read more → langchain.com/conceptual-gui…
Sam Crowder retweeted
Environments in LangSmith Prompt Hub
Environments give you a proper promotion workflow for your prompts:
- Assign any commit to Staging or Production
- Promote between environments instantly
- Roll back with a single click from a full deployment history
- Reference reserved tags (`staging`, `production`) in your code so the right version is always served with no code changes needed
The same promote-and-rollback pattern you already use for app deployments, applied to prompt management. Give it a spin :rocket:
Docs: docs.langchain.com/langsmith…
Sam Crowder retweeted
💫 New LangChain Academy Course: Building Reliable Agents 💫
Shipping agents to production is hard. Traditional software is deterministic – when something breaks, you check the logs and fix the code. But agents rely on non-deterministic models.
Add multi-step reasoning, tool use, and real user traffic, and building reliable agents becomes far more complex than traditional system design.
The goal of this course is to teach you how to take an agent from first run to production-ready system through iterative cycles of improvement.
You’ll learn how to do this with LangSmith, our agent engineering platform for observing, evaluating, and deploying agents.
Enroll for free ➡️ academy.langchain.com/course…
Sam Crowder retweeted
Open Models Panel at GTC with Harrison & Jensen:
Join us tomorrow, Wednesday March 18th at 12:30pm at GTC for “Open Models: Where We Are and Where We’re Headed”, a panel featuring Harrison, Jensen, and the CEOs of Cursor, Thinking Machines Lab, Perplexity, and more.
Add it to your schedule ➡️ nvidia.com/gtc/session-catal…
Sam Crowder retweeted
New Conceptual Guide: You don’t know what your agent will do until it’s in production 👀
With traditional software, you ship with reasonable confidence. Test coverage handles most paths. Monitoring catches errors, latency, and query issues. When something breaks, you read the stack trace.
Agents are different. Natural language input is unbounded. LLMs are sensitive to subtle prompt variations. Multi-step reasoning chains are hard to anticipate in dev.
Production monitoring for agents needs a different playbook. In our latest conceptual guide, we cover why agent observability is a different problem, what to actually monitor, and what we've learned from teams deploying agents at scale.
Read the guide ➡️ blog.langchain.com/you-dont-…
