We have some fun voice AI stuff to talk about on the @NVIDIAAI Nemotron Labs livestream tomorrow: fine-tuning audio and LLM models for specific agents and use cases. Join us live if you want to chat and ask questions. 10:00am Pacific Time on Tuesday.

More multi-model voice AI demos from Jon: this time a tiny bi-directional encoder model in the Pipecat pipeline that runs on CPU fast enough to be in the main voice loop. Fun fact: you can train a classifier model like this in less than 10 minutes on a modern GPU. (If you have the right data set. All ML problems are data problems.)

We blew past the Turing test for voice agents about a year ago. I don't think we talk about this enough. We have lots of data from real-world voice agent deployments. In contexts like customer support, people often forget they are talking to a voice agent. This is true even when the system tells people they are talking to an AI agent at the beginning of the session. (Which I think is a best practice that almost all AI agents should adhere to.) So ... when will this happen for realtime, conversational AI video? Well, we're getting very close. @tavus just released a new realtime video model that, at its best, passes the Turing test for conversational video. It's pretty amazing. The model is full-duplex (it can talk and listen at the same time), and does vision and audio input and audio+video output in realtime. Both on the nascent conversational video benchmarks, and in my testing, it's a pretty big step forward. Here's an unedited video of one of my conversations with the new model. I think you can see a few interesting things here. When the model responds as quickly as you expect a human to respond, the experience is magical. The responses are conversational and natural; there's no leaning on stock filler phrases to hit the response time goals. The integration between the video and audio feels natural, which has historically been an uncanny valley thing that most video avatar experiences didn't quite get past. The model can do hand gestures! If you're deep into this stuff, you'll find that a little bit mind-blowing. And, in keeping with all the stuff that Tavus has released, the fact that the model can "see" as well as hear and talk is integral to the experience and a big part of what makes the conversation feel useful and interesting. At the very beginning of the conversation, you can see that the model and I talk over each other. Which is something that happens a lot in human-to-human video call conversations. We both recover from this gracefully, which, again, has been something that rarely worked quite perfectly in previous generations of realtime video systems. Even more than the "when it works perfectly" capabilities, it's probably these recovery and robustness moments that get us all the way to experiences that feel completely natural.

Here's a complete voice agent tutorial focusing on a super low latency configuration: Pipecat PhoneLLM Alpha 1 on @modal, Deepgram transcription, Cartesia voice. You can grab the code and jus run /setup in Claude or Codex. But Merve's video also dives into the details of configuring model endpoints on Modal, the Pipecat bot file, and the front-end UI code in the example project.

Everybody building voice agents obsesses about latency. Every enterprise I talk to wants to run LLMs on their own infrastructure (for data management, regulatory/compliance, and cost reasons). PhoneLLM is a new open weights model trained on voice agent tasks: sub-100ms inference-side TTFT 🤝 ability to run a highly capable LLM in your own cloud. I went on @ThursdAI last week to talk about PhoneLLM, voice agent latency and cost, and *why* all the LLMs released in the past two years are too slow for voice agents (it's all about test-time compute).

Introducing PhoneLLM, an open model for voice agents. GPT 5.6 Terra performance on typical voice agent tasks at 1/3 the latency and 1/18 the cost. For voice agents, we need models that are both very low latency and very good at tool calling and instruction following. There's a trade-off here, and we often have to compromise on either latency or capability when building voice agents. With PhoneLLM (and the training and data stack that made this model possible) we're fixing this problem. For the last couple of years, most of the effort in frontier model development has gone towards leveraging test-time compute. Which is awesome! Models of all shapes and sizes are available that perform really, really well ... if you have "thinking" turned on for your model. But if you need your agent to respond at voice conversation speed, you can't use thinking models. PhoneLLM is a full-weights fine-tune of NVIDIA Nemotron Nano 30B. We trained on a wide range of real-world telephone and customer support use cases. The training focused on taking the excellent Nano 30B base capabilities and teaching the model to do typical voice agent tasks with thinking disabled. The results are really good: accurate tool calling and concise, on-topic responses in long conversations. And fast: TTFAT measured server-side is <100ms if you run PhoneLLM on a lightly loaded B200. :-) But seriously, when we characterize model latency, we do it with full, end-to-end, batched request simulations using real Pipecat voice agent pipelines. You can serve more than 80 concurrent agents on a single B200 with P95 end-to-end TTFAT <600ms. Including network overhead. That's an LLM cost-per-minute around $0.0025. (1/4 of a cent.) At a latency lower than any third-party API offers today. More details about this model, including weights on @huggingface, how to spin it up with one click on Modal, and a starter project repo you can clone, are in the thread ...

Pipecat v1.8.0 today, with launch day support for Gemini 3.5 Transcribe, Google's new Gemini-based transcription model. This is a very big Pipecat release. There are 233 entries in the changelog. Some highlights: 🛡️Improvements to error handling and service failover: Processors can now report when they are no longer usable, errors are categorized, and ServiceSwitcher only fails over when a service really can't recover. STT/TTS services also stop endlessly retrying permanent failures like invalid API keys or models. ⚡ Faster pipeline startup: Pipeline setup has been reworked around the new setup() lifecycle, allowing processors and services to initialize and connect concurrently before the pipeline starts. Combined with import-time improvements, this significantly reduces startup time, especially for larger pipelines. 🗣️ More flexible turn management: Services with built-in turn detection now propose turn boundaries, while Pipecat's turn strategies make the final decision. This keeps turn management and interruption handling in one place and makes provider-native turn detection much easier to customize. 🧪 Pipecat Evals keeps getting more powerful: Run scenarios during development to iteratively measure pass rates, get machine-readable results.jsonl, evaluate turns individually, run entire directories of scenarios, and more. Great for measuring nondeterministic behaviors like interruptions and async function calls instead of testing them once and hoping for the best. 🤖 Better coding-agent experience: Pipecat Context Hub is now included with the CLI and integrated into pipecat init, including automatic setup for supported coding agents and freshness checks to help prevent agents from generating code against stale Pipecat APIs. 🔧 Function calls can now be cancelled by the LLM: Long-running async tools can opt into LLM-driven cancellation, and timed-out function calls are now automatically cancelled instead of continuing to run in the background. 🔌 New Speechify TTS, ElevenLabs Dialogue TTS, Bland TTS, Sarvam Realtime STT, Deepgram Flux on SageMaker, OpenClaw Gateway support, LiveKit runner support, MoQ client mode for connecting bots through a relay, including deployments behind NAT and more. And, of course, there are tons of fixes and smaller improvements throughout the framework, including pipeline startup/cleanup, metrics, TTS tracking, transports, realtime services, and turn handling. Huge thanks to everyone in the community for making everything we do possible! There are now more than 180 Pipecat integrations. ❤️ Happy hacking! 🚀

I spend a lot of time on benchmarks, these days. I help maintain several public benchmarks, work with customers on private benchmarks, and talk to partners about how we can all build benchmarks that are useful for evaluating all of the components of our voice agents. This benchmark by @_josh_meyer_ and team is carefully constructed to measure the things that are critical to voice agent success, in a scenario that closely matches the real-world use case we see in enterprise deployments. A few notes, from my perspective, in praise of things I really appreciate about this benchmark work: - A good voice agent benchmark tests multi-turn conversation, with multiple defined tools and several tool calls during the conversation. - Users care about both overall conversation success and the turn-to-turn naturalness and latency during a conversation. It's important to do a good job quantitatively scoring both of these. You can aggregate them into a single score, or break out the components (overall success, voice-to-voice latency, unwanted interruptions, etc). - Teams building agents want to see comparisons between cascaded pipelines (TTS -> LLM -> STT) and speech-to-speech models. A good benchmark can judge both architectures on the same tasks. - Cost matters. Most teams think about cost as cost-per-minute. Translating token counts to cost-per minute is note easy. And even forecasting token counts is hard to do in the abstract. A realistic benchmark can generate token and caching data that helps pin down the cost of different models and architectures. - Tool call failures (and other types of agent failures, too) can be subtle. When you're working on a benchmark, it usually takes a fair amount of iteration and scrutinizing the transcripts to get to the point where your LLM-as-a-judge is doing a good job judging. - Open source benchmarks and detailed write-ups allow other people to learn from the hard work that goes into benchmarks, and force you to stand behind the choices and implementation quality of your benchmarking work. Benchmarks are hard! It's good to build them in the open.

The unconference sf events that @vaibcode and @dexhorthy host are amazing. Last October at unconf was the first moment it really clicked for me that all the things we'd built to make production voice agents work had exact parallels in basically all the other categories of agent workflows. There were a bunch of great presentations (with code and with real, interesting examples of hard stuff and corner cases) on subagent management, loop and graph control, context sharing across different agents and different LLMs, long long long long context engineering, structured data, LLM as a judge, and more. The unconf this Spring was also great. There's another one coming up on October 12th. If you're in SF and have a densely technical talk that you think other people building AI-native software would learn from, come hang out. We put talk titles up on a whiteboard and the person who just gave a talk picks the next one!

If you're in San Francisco on Thursday, come hang out with us and talk about Agents, APIs, designing directly in code, multi-model routing, and AI-native software. We're starting to write new kinds of software, built from the ground up around LLMs. (I don't mean building with the help of LLMs. That's true too, of course, in a big way. I mean software with LLMs and inference calls happening in every layer of the stack, all the time.)

Lots of new stuff in recent Pipecat releases. There are 245 items in the CHANGELOG.md entries for the three releases so far this summer. Pipecat is the most widely used voice and video AI framework, and supports a very large range of use cases. So opinions will differ about what's most important in each release! To me, the three biggest new, new things in Pipecat recently are: 1. Subagents 2. Folding Pipecat Flows into core 3. A new behavior evals framework Subagents. We built Pipecat to power realtime, human-in-the-loop AI software. The first-generation voice agents and voice assistants that showed people what's possible with LLMs+voice are just the beginning. Most of our customers' production deploys today, and all of our experiments and side projects, now use multiple LLMs and multiple inference loops. The LLM is the new function. The inference loop is the new thread. The new `TaskManager` and `Workers` abstractions in Pipecat provide support for starting, stopping, and managing these subagent "threads." Subagents share LLM context using a message bus. The bus can be local or can serve multiple agents running on different machines. Delegating work to subagents keeps the conversational voice loop fast, relatively simple, and non-blocking. Common uses of subagents include managing communication with legacy enterprise systems, implementing guardrails, and dynamically generating on-screen user interfaces. Pipecat Flows. One of my mantras is that everything is a state machine. Many production voice agents are explicitly designed as state machines. Conversation phases with well-defined transitions between them. Think about the different "jobs to be done" during a healthcare appointment pre-visit call: authentication, time/date confirmation, confirmation of reason for the visit, pre-existing conditions and current medications, etc. At each transition point, we swap out the system instruction, generate a new tools list, and do a (non-blocking) contextually aware summarization of the context history. Pipecat Flows is a set of helper modules for implementing these flexible but robust conversational state machines. When people are having trouble getting their complex voice agents to score well on "task success" evals, the single most powerful lever is Pipecat Flows. New evals support. We've always worked closely with evals and devops platforms, building strong support in Pipecat for all of the widely used tools in this space. And we strongly recommend that every enterprise that is deploying voice agents at scale explore the evals/tooling landscape and pick a vendor to work with. But early in the development process, you get huge leverage from simple automation of smoke tests, end-to-end tests, and basic behavior testing. If you have a lot of time in the saddle building both voice agents and working with coding agents, you know how to prompt these automations into existence. The new Pipecat evals framework makes this easy for new developers, too, and makes this workflow trivially "visible" to coding agents. I'll leave you with a little bit of history, because I've been talking about this a lot, lately. A bunch of us who work on Pipecat have been building this "new new" human-in-the-loop stuff together for a long time, across two startups. We worked on the user interfaces in Minority Report and Iron Man. LLMs+voice are giving us the final, missing pieces we need to build the things we've been dreaming about and working on since ~2005 or so. Here's a demo video from 2012. Pre-LLMs. Pre-Pipecat. But, for us, the same long project!

Microsoft announced a bunch of interesting new AI models and tools this week. Model launches alway get lots of attention. But don't sleep on the new ASSERT evals framework that launched today. I'm on record as arguing that 2026 is the year of evals. Evals are the glue for all the "jobs to be done" at every level of AI: model training; testing and deciding on what models to use and how to use them; and testing and improving AI agents in production. Evals unify our work on those different layers of the stack. These days, when we talk about evals, observability, and testing, we're talking about overlapping parts of a large set of tools we're still early on in figuring out. As the AI engineering ecosystem matures, diversifies, and increases massively in scale, we really, really need good evaluation (observability, monitoring, testing, data management) frameworks. I got a chance to test the new Microsoft ASSERT evals framework before it was released, and it has some very nice core ideas. 1) ASSERT is open in two important ways. First, the team is serious about broad support for models, frameworks, and use cases. Microsoft spent time understanding voice agent use cases and building Pipecat support, for example. Second, the code is completely open source, released under an open MIT license. 2) We're all working in and with agentic coding tools today. That means we are planning in natural language, and all of our software development and ops tools have to evolve for these new, natural language, workflows. ASSERT takes descriptions of desired agent behavior and generates specifications for the ASSERT suite of tools to run against. In a world where "English is the programming language," how we actually make natural language "code" precise enough and repeatable enough is perhaps the big unsolved tooling problem that all of us are working towards in different ways. This is true whether we work on coding agents, AI opps tooling, orchestration frameworks, or vertical applications. 3) Microsoft describes ASSERT as a policy-driven framework. Rather than eval against generic performance metrics, ASSERT aims to generate stable but adaptable evaluation criteria for specific agents. "Policy-driven" also implies a full loop design. Policy (generated from specific requirements) -> evaluation -> optimization -> monitoring in production -> improving the policy description -> evaluation -> ... 4) Enterprise agents need to be evaluated along many dimensions: task completion, individual conversation turn behavior, latency, mode-specific metrics like audio disfluencies, and safety/security. Microsoft designed ASSERT to be used together with a new safety governance toolkit called Agent Control Specification. 5) Finally, ASSERT is integrated into the Microsoft Foundry ecosystem. Today, AI engineering tools have to be open source and vendor neutral to get attention from developers and gain widespread adoption. *And* it's equally important to give enterprise customers tools that work as a coherent stack. This is hard to do well. There are real tensions between open source development versus engineering a great full stack developer experience. However, if you sweat the details on both ends, you benefit from a full spectrum of feedback about real-world development pain points. It's more work, but it's worth it! Kudos to Microsoft for embracing this and committing to an open, community oriented approach, plus doing the extra work to build the full stack for enterprise customers.

Voice agents hackathon at @ycombinator in SF on May 30th. Prizes include a guaranteed YC interview, and special awards from sponsors @cekuraAi, @NVIDIAAI, @AWS, and @twilio. Learn to build agents that work at scale, in production. Use tooling from Cekura to simulate and auto-improve your agents. Handle accents, noisy environments, interruptions, and customers who don't follow the expected script! Build with NVIDIA Nemotron open source models, running on AWS infrastructure. Integrate with Twilio's telephony platform. Leverage the Pipecat developer ecosystem. Join us for fun, learning, conversations with the engineers building all of the above tools, food, and prizes.

I care a lot about 1) latency, and 2) good benchmarks. The voice models from @GradiumAI occupy in the top spot on the @covaldev voice AI benchmark leaderboard, which tests both latency and accuracy. We've made the Gradium models the default in our open source, massively multi-player, LLM game. You play the game by talking to your "ship AI." Of course, humans aren't the only talkers in the universe, these days. So @chadbailey59 has been building OpenClaw bots that play the game, too! Here's a video of an OpenClaw agent playing the game by talking to the agent's ship AI. Chad created specific personalities for both the OpenClaw bot and the ship AI, here. Gradium's voice models are very emotive, and they support voice cloning and customization. I love how morose and pessimistic the ship AI is, and how excitable and cheerful the OpenClaw bot is. This is not scripted at all. These are two voice AI agents just doing their thing.

Last week at the SF Voice AI Meetup, I moderated a panel about multi-modal model training, with Jagadeesh Balam who works on speech models at @NVIDIAAI, Fabian Seipel of @ai_coustics, and @code_brian from Tavus. I always really enjoy the opportunity to hear from people working on models (small, large, text, audio, pixels, transformer-based, diffusion, etc)! Some notes: - Brian said "latency is solved," if you're thinking about latency as a mechanical problem. Humans take ~700ms to think about things before they respond in conversation. Current STT->LLM->TTS pipelines can beat that. What's missing is the higher-level architecture for "thinking": queuing what to talk about next, deciding what to say first and how, tracking emotional tone, etc. - Jagadesh said that as we do more and more interesting things with the models, the bar for performance goes up. Transcription was "solved" for non-realtime use cases, but now voice agents need fast and accurate transcription of very tricky strings like email addresses and mixed alphanumeric account numbers. And for speech-to-speech models, we have to clear the bar of performing well in long, multi-turn conversations. Part of the challenge here is generating very good training data. "Data simulation for training is unsolved. If it were solved, all our model roadmaps would be done by now!" I appreciate this viewpoint, because I don't think we talk enough about the challenge of having large amounts of *exactly* the right training data. - Fabian talked about how ai|coustics generates data for training very fast, very specialized audio models that improve the performance of voice agents. His team includes people who spend a lot of their time simulating room geometries, mic frequency responses, WebRTC processing artifacts, and many other things. He calls them "professional audio destroyers."

✨ Voice AI, open models, and next-generation evals hackathon at @ycombinator in SF on May 30th. ✨ We're co-hosting with @cekuraAi , and we've pulled in our friends at @NVIDIAAIDev, @AWS, and @twilio for expertise and mentoring. We'll help you build state of the art voice agents using: - NVIDIA Nemotron models - AWS SageMaker and Bedrock inference - Twilio telephony - Cekura evaluation tooling - Pipecat orchestration and Pipecat Cloud agent hosting Up for grabs: - A guaranteed YC interview - Special judges' prizes from NVIDIA, AWS, and Twilio for the most impactful and technically impressive projects Join us to learn from engineers who built all the tools you're using, compare notes with other voice AI developers, and show off your ideas! Space is limited. Apply below.

Voice AI turn taking is a solved problem. The single most common complaint about voice AI, today, is that agents interrupt too often. But the voice agents I build for myself now respond quickly and interrupt me less often than the people I talk to every day. (I actually measured this.) @mark_backman made a @pipecat_ai PR two weeks ago that was the last piece of the puzzle for turn taking so good that I no longer ever think about it. The approach combines three layers of processing: 1. Voice activity detection, with a short (200ms) trigger. 2. A native audio turn detection model that's small, fast, and runs on CPU. This model captures audio nuances like inflection and filler sounds that don't get transcribed. 3. A prompt mixin for the conversation LLM that decides turn completion based on conversation context. None of these are new. We've been using VAD for a long time. We trained the first version of the Pipecat Smart Turn native audio model in December 2024. And we've been experimenting with prompt-based large model turn detection (sometimes called "selective refusal") for more than a year. Now, the Smart Turn model and the SOTA LLMs we're using in voice agents have both gotten so good that using them together feels like we've finally "solved" turn detection. Mark also figured out how to elegantly apply a "single-token tagging" technique to this problem. We sometimes use single-token tagging in place of tool calling, when we need a near-zero latency programmatic trigger. Mark's Pipecat mixin defines three single-token characters and prompts the LLM to output exactly one of them at the beginning of every response. - ✓ means the agent should respond normally (immediately) - ○ is a "short incomplete" - the agent should wait 5 seconds - ◐ is a "long incomplete" - the agent should wait 10 seconds The wait times, and the details of the prompt, are configurable, of course. Watch the video to see me talk to an agent that handles all my various pauses and inflections, plus phrases like "let me think," pretty much the way a person would handle them, in terms of response latency. Also, in the second half of the video, I ask the agent to adjust its response pattern because I'm going to tell it a phone number. This kind of "in-context" adjustment of response wait times is really useful. The LLM in the video is GTP-4.1. We've tested the prompt and single-token adherance with GPT-4.1, Gemini 2.5 Flash, Anthropic Claude Sonnet 4.5, and AWS Nova 2 Pro. Note that older models in all these families (and, in general, smaller open weights models) aren't able to reliably output these single-token tags. But the new models we're using these days are pretty amazing.