we just added the ability to assign specific reviewers to an annotation queue. as building agents becomes more and more automated, human feedback being part of the loop will be as critical as ever.

the more things we ship, the more our customers use LangSmith, and the more resources they create within the product we just made it easier to see and page through all these resources. new table designs out today!

we added the ability to apply an evaluator to existing experiments. we're seeing more and more customers use experiments as long term sources of truth, mainly as a baseline to compare against. as you define new evaluators, you want to be able to see how they compare against past runs

we shipped a new version of our monitoring tab that enables navigating more easily between dashboards and alerts, and brings the charts up one level of depth in the product would love to hear any and all feedback!

3% of the time it hallucinated success when it actually hadn't completed the task. Yikes!

4.5% of the time clawdbot ran into task persistence failure, failing to save the task result to a local filesystem that it had designed for itself.

Give the simplicity of the tasks, 64% completed successfully! Go clawdbot!

After running a random task on my machine every minute (find me a flight to New York, check availbility of Korean restaurants in SF, etc) I had about 100 traces to work with! I then ran a LangSmith Insights job over those traces with a prompt to group traces by failure mode.

With agents, code does not define behavior -- traces define behavior. To peak into what clawdbot was doing on my computer, I hooked up tracing to LangSmith: