Marco

Langchain released a feature that clusters agent traces by behaviour patterns. I just ran it on 500 production traces. Here's what it actually does. It's an LLM analyzing your LLM traces. Specifically: 1. Takes your production traces (inputs, outputs, tool calls, intermediate steps) 2. Runs them through clustering based on semantic similarity 3. Uses an LLM-as-judge to categorize and label clusters 4. Generates a hierarchical taxonomy of usage patterns The interesting part is the clustering methodology. From the UI, you can see that it's extracting intent from the full traced context. This is applied across all traces, then similar intents get clustered. Each cluster gets labelled both by pattern type and by quality metrics. What I've found useful here is that each subcategory represents a distinct agent system and context engineering pattern. RAG retrieval, for example, has different failure modes than web scraping, even though both are "information sourcing." The system is essentially discovering that your agent has multiple operational modes, each with different context & user requirements. So now, this means I can fix categories of failures, not just individual bugs. What I really like about Langsmith and their new Insights Agent is the configuration options. You can customize clustering dimensions: - Group by usage patterns: Clusters based on what users are trying to accomplish - Group by poor interactions: Clusters based on failure signatures (user frustration signals, error patterns) - Custom attributes: Define your own dimensions for clustering Some experiments with custom attributes that I'm building now: - context_switches: Number of topic changes mid-conversation - tool_chain_length: Number of tool calls in sequence - retry_rate: How often the agent needs multiple attempts This is a great feature for strategic analysis (daily/weekly reviews). My first impression: the categorization is surprisingly good. Clusters are semantically coherent. The metrics are well-chosen for diagnosing agent issues. Useful for understanding agent behaviour at scale. Much better than manual review because it finds patterns you wouldn't notice in individual traces.