
Building RTC/AI services @Tavus. ex-@RingCentral @EpicGames @Houseparty @Telefonica
Spain (but who cares)
Joined May 2007
- Tweets5.2K
- Following475
- Followers2.2K
- Likes7.5K
Voice Agents making calls answered by other Voice Agents
Claude Code reviewing code written by Claude Code
If that's the future (no doubt) we should at least optimize the tools for them to communicate instead of having to use voice and github :D
Weekly Voice and Video AI Update 🎉 👇
The speech-to-speech race is heating up. xAI shipped Grok Voice 3.0, which jumped to #1 on Artificial Analysis' speech-to-speech index. OpenAI answered with GPT-Live-1.1, which adds 13 new voices and better accents, and opened prompt-to-voice to every API customer. Amazon also made Nova 2.5 Sonic generally available on Bedrock, with better reasoning, async tool calling and a 256K context window at the same price as before.
And again more news on the video avatars space coming from Tavus with Kiosk mode and Telnyx AI Assistants. Also a trial a voice companion for snowbound seniors coming from ElevenLabs and NTT that looks like a great use case for video avatars in the future.
The infrastructure is catching up just as fast. LiveKit Inference now runs cascaded, speech-to-speech and full-duplex models side by side. Turn-taking is still where a lot of the work is. ElevenLabs added turn prediction to its real-time STT, Pipecat shipped user and bot backchannels, and a great webrtcHacks piece shows how leaked echo still tricks agents into hearing turns that never happened.
FULL UPDATE IN THE NEWSLETTER LINK BELOW 👇
Have a great week!
Trying Jev but in a different way.... Using it now to detect Region of Interest in a basketball game and also running it on-device with Tev1 implementation instead of Typesafe's. Looking good so far and makes my life much easier :)
🟡 Yellow is my previous algorithm: YOLO + OC-SORT + heuristics
🟣 Purple is Tev1
No menus to tap through. You just walk up and talk.
Kiosk mode puts a Tavus PAL on a walk-up kiosk. Here it’s checking in a guest at our front desk.
We’re opening a pilot to a limited number of companies. Want in? Let us know here: tavus.io/demo
I wanted to do some experiments with @livekit + @tavus to learn more about LiveKit Agents framework and I decided to reimplement the awesome Insurance Claim Agent example from @Saboo_Shubham_ but using LiveKit Agents and Tavus for more natural face expressions.
You can find the code in the github repo link below, including some diagram explaining how to combine all the pieces:
- Live voice implemented with LiveKit Agents framework and OpenAI voce models
- Realistic face with natural expressions from Tavus Phoenix 4.5 via its LiveKit plugin
- Claim extraction in the background with async tools
- Camera evidence captured and checked by a vision model
- Incident sketches drawn as SVG by an LLM (I love this part from original Shubham's demo ♥️)
Bonus: We always write agents in python and it's becoming boring so this time I decided to use the typescript version of LiveKit Agents (anyway CC wrote most of the code :P)
Gustavo Garcia retweeted
I spent 48 hours in SF hanging out with billionaires, data labs and AI founders and execs running $1B to $100B companies.
My observations:
1. Sooo MANY stories of data brokers making like $2M a WEEK selling company data to labs. I can't tell if it's the beginning or a bubble, and I kept wondering which niches are left. Digging into it actively.
2. At dinner, someone asked the table, "If a new model drops next month, does your company get better or worse?" HALF the table went quiet! I think that's the best question to ask about any AI startup right now.
3. The bar people talk about now is hitting $1B in revenue in 12 months (I guess that's the new unicorn?). What I didn't hear much about is retention. AI revenue can come in fast and leave fast, and I think we're going to see a lot of stories of AI companies with huge revenue and terrible retention.
4. It's pretty clear we're living through the reinvention of the computer. The model is the CPU, the harness is the operating system, and skills and plugins are the apps. Thinking about it that way makes it easier to see where the whitespace is.
5. We're maybe 12 months from an agent that can watch you work for a week and then do your job without any instructions. The screen recording plus agent memory plus local model combination makes this possible right now.
6. Agent harnesses feel like the new GPT wrappers, in a good way, especially vertical ones. The best teams I talked to treat the model as swappable. One founder walked me through routing different models to different jobs: one for taste and visuals, a cheaper one for the everyday stuff, another for text.
They use prompts they can move across models over fine-tuning, and A/B test the output on what users actually do with it: do they edit it, export it, share it? A vertical harness is basically four things: the connectors for that industry's data, evals written by people who know the work, rules for what the agent can and can't touch, and a clean screen for a human to review. The model is maybe 20% of it.
7. There's a lot of reverse engineering going on/cloning going on more than I've ever seen. People are decompiling software and rebuilding it with agents, and someone showed a suite of like 10 apps that are worth $5B+ each all that he cloned open source. Me and everyone are still thinking through what it means when software gets this easy to copy.
8. My guess is the value moves to data. If the software can be rebuilt quickly, what's worth paying for in a legacy SaaS company is the years of customer data and workflows inside it. Some SaaS companies are secretly data companies.
Gets me thinking about how do you truly value data for this saas companies? Like if we were buying these dying saas companies at 2x ARR before (as example), maybe there is a better metric. Idk, just thinking out loud here.
9. It feels like a good chunk of legacy software won't make it to the agentic era, especially anything built around people clicking through a lot of screens.
10. Every company I talked to wants to pipe its internal data into one place and expose it to agents through MCP. Then they look at the data and realize it's a mess: five versions of the same customer, old Slack channels, PDFs nobody can search. Cleaning that up feels like a big consulting opportunity!
11. A lot of founders are building the basic building blocks for agents: APIs, data layers and tools. Some of the companies I saw barely have a UI because the main user is an agent. I really believe the startups that build primitives to this agentic era will do extremely well. Building blocks is the best way to think about it.
12. Labs can get plenty of training data, but it's harder to get real examples of agents failing inside companies. Whoever has those logs has something valuable.
13. Someone running a big consumer company told me the next fight is over memory. The model that remembers the most about you wins, and the smartest model matters less than people think.
14. Physical AI came up a lot. Astra I think was the first model that got people into AI can actually do CAD, and I think a lot of people are thinking what does vibe manufacturing really look like. The line I keep thinking about came from a robotics founder: the hardest problems in AI are deployment problems now, not research problems. Their robots work on their own, call a human when they get stuck, and every one of those calls becomes training data.
15. Cybersecurity companies are having a moment and are actually cool companies to build now, especially the ones that clean up after a breach. Every agent with access to your systems is a new door into your company, and a lot of those doors were left open.
16. Some startups are worth more dead than alive now. A shutdown company with 5 years of Slack, email and code can get paid for licensing all of it. I wouldn't be surprised if founders start factoring that into when they shut down.
17. People are starting to talk about monetizing the agent web. It feels really early, but when the most popular user of the internet is an agent, what startups should you be building?
18. I think agent insurance is coming. Once an agent makes an expensive mistake, insurers will want audits before they cover anyone.
19. Several people said the hire they want most is someone with taste who can write evals. The models are getting good at generating ideas, so the hard part is judging which ones are good.
20. New trend I observed: startup raises VC, uses venture dollars to become a creator, startup fails, founder falls back on being a full-time creator and incubates other products bootstrapped.
21. Getting SIGNIFICANT distribution is HARD right now. Founders are sponsoring buses, buying billboards like crazy and throwing parties with famous DJs to get noticed. Kinda feels dotcom era in that sense.
Attention is what's scarce! The growth guy going viral sharing his UGC strategy from Wispr Flow is a good example. It's the sauce everyone wants, and I truly believe there are probably only 50 people on the planet who know how to scale these types of businesses efficiently in this era.
Always good to get that dose of SF. I felt sooo lucky to have such an incredible trip during such an incredible time in building businesses. Surreal in so many ways.
Happy to be home and locked in.
Until next time!
Voice AI Weekly Update 🎉!
The market keeps consolidating: @inworld acquired @ultravox_dot_ai to double down on speech-to-speech, while Hume is shutting down its TTS and EVI APIs in November. And @ElevenLabs doubled its valuation to $22B with a $300M employee tender, with agent conversations up 3x since February.
This week AI video crossed a line I didn't expect to see so soon. @tavus introduced Griffin, a full-duplex video-to-video model, and 48% of people who talked to it live for a minute thought it was a real person 🤯.
On the speech side 🗣️, @Microsoft shipped its first streaming transcription model, MAI-Transcribe-2-Streaming, which went straight to #1 on Artificial Analysis at $0.54/hr, alongside new MAI-Voice 2.1 TTS models. Also many frameworks and solutions are moving ahead and integrating @OpenAI GPTLive1 for more natural conversations.
Take a look at rest of updates and the Readings and Releases section in the full newsletter below 👇. Have a good week!
Great post from Philipp @HCornflower 👏. "This is why this fits in 0.16% of a cpu core" 🤯. The diagrams and experiments were very interesting.
We still need to figure out how to tune AEC for Voice AI though :)
New post: Google Chrome downloads a new neural AEC while you are not looking
@HCornflower reverse engineers the model, tests it against AEC3, and asks why other browsers don't get the weights.
webrtchacks.com/chrome-neura…
Few people have spent more time working on real-time video than Kwin, so this means a lot.
Thanks for testing Griffin early and sharing your experience and feedback!
We blew past the Turing test for voice agents about a year ago. I don't think we talk about this enough.
We have lots of data from real-world voice agent deployments. In contexts like customer support, people often forget they are talking to a voice agent. This is true even when the system tells people they are talking to an AI agent at the beginning of the session. (Which I think is a best practice that almost all AI agents should adhere to.)
So ... when will this happen for realtime, conversational AI video? Well, we're getting very close.
@tavus just released a new realtime video model that, at its best, passes the Turing test for conversational video. It's pretty amazing.
The model is full-duplex (it can talk and listen at the same time), and does vision and audio input and audio+video output in realtime. Both on the nascent conversational video benchmarks, and in my testing, it's a pretty big step forward.
Here's an unedited video of one of my conversations with the new model.
I think you can see a few interesting things here. When the model responds as quickly as you expect a human to respond, the experience is magical. The responses are conversational and natural; there's no leaning on stock filler phrases to hit the response time goals. The integration between the video and audio feels natural, which has historically been an uncanny valley thing that most video avatar experiences didn't quite get past. The model can do hand gestures! If you're deep into this stuff, you'll find that a little bit mind-blowing. And, in keeping with all the stuff that Tavus has released, the fact that the model can "see" as well as hear and talk is integral to the experience and a big part of what makes the conversation feel useful and interesting.
At the very beginning of the conversation, you can see that the model and I talk over each other. Which is something that happens a lot in human-to-human video call conversations. We both recover from this gracefully, which, again, has been something that rarely worked quite perfectly in previous generations of realtime video systems. Even more than the "when it works perfectly" capabilities, it's probably these recovery and robustness moments that get us all the way to experiences that feel completely natural.
Gustavo Garcia retweeted
Glad I can share this finally! If you are not impressed by these demos and numbers of Griffin I don't know what else you need. I recommend you the Simon Says one below.
It happens to me often in IG that i'm not sure if a video is real anymore but I didn't think we were so close to human experience IN REAL TIME 🤯.
I dare you show your friends and check if they can distinguish real humans from Tavus digital humans. And this is just the beginning.
Introducing Griffin, the first model to pass the video Turing test.
48% of people who talked to it live thought it was a real human. Previous systems have had a pass rate <3%. It is #1 on NVIDIA's benchmark for full-duplex AI video.
It’s the first Human Interaction Model (HIM).
Readers added context they thought people might want to know
The 48% figure and "video Turing test" claim are from Tavus's own study of 54 one-minute calls, not independently verified or using a standard protocol. Griffin-Lite leads NVIDIA's VideoFDB benchmark on their public leaderboard.
cellcog.ai/blog/tavus-gri…
research.nvidia.com/labs/amri/proj…
tech-ish.com/2026/10/02/tav…
Glad I can share this finally! If you are not impressed by these demos and numbers of Griffin I don't know what else you need. I recommend you the Simon Says one below.
It happens to me often in IG that i'm not sure if a video is real anymore but I didn't think we were so close to human experience IN REAL TIME 🤯.
I dare you show your friends and check if they can distinguish real humans from Tavus digital humans. And this is just the beginning.
Introducing Griffin, the first model to pass the video Turing test.
48% of people who talked to it live thought it was a real human. Previous systems have had a pass rate <3%. It is #1 on NVIDIA's benchmark for full-duplex AI video.
It’s the first Human Interaction Model (HIM).
Readers added context they thought people might want to know
The 48% figure and "video Turing test" claim are from Tavus's own study of 54 one-minute calls, not independently verified or using a standard protocol. Griffin-Lite leads NVIDIA's VideoFDB benchmark on their public leaderboard.
cellcog.ai/blog/tavus-gri…
research.nvidia.com/labs/amri/proj…
tech-ish.com/2026/10/02/tav…
Gustavo Garcia retweeted
🗞️ Tons of Voice and Video AI updates this week! I'll try to keep it short here, but you can find the full update in the newsletter link below.
I would call last week the week of Realtime Avatars: @GoogleAI included support in Gemini Live 3.8, @Meta's personal AI agent Muse got digital avatars, and leading avatar provider @tavus released its Phoenix 4.5 version enabling even more natural conversations. On top of that, Unit1 raised $20M to bring real-time avatars to concert stages! 🎸
Also interesting hardware news around Glasses and Earbuds from @Meta and @Qualcomm and the ability to make automated phone calls from @GeminiApp and Grok @bot.
On the speech models side, we got Eleven v4 from @ElevenLabs, @AlibabaGroup released new cheaper models from the Qwen Audio 3.1 stack, and @nvidia 's open-sourced diarization model, among other updates.
The frameworks teams stayed very busy, with @livekit releasing a Private Link feature to integrate enterprise infrastructure into cloud agents, and @pipecat_ai adding support for the cool new kid, JEV, with its new Classifier capabilities.
Have a great week!
🗞️ Tons of Voice and Video AI updates this week! I'll try to keep it short here, but you can find the full update in the newsletter link below.
I would call last week the week of Realtime Avatars: @GoogleAI included support in Gemini Live 3.8, @Meta's personal AI agent Muse got digital avatars, and leading avatar provider @tavus released its Phoenix 4.5 version enabling even more natural conversations. On top of that, Unit1 raised $20M to bring real-time avatars to concert stages! 🎸
Also interesting hardware news around Glasses and Earbuds from @Meta and @Qualcomm and the ability to make automated phone calls from @GeminiApp and Grok @bot.
On the speech models side, we got Eleven v4 from @ElevenLabs, @AlibabaGroup released new cheaper models from the Qwen Audio 3.1 stack, and @nvidia 's open-sourced diarization model, among other updates.
The frameworks teams stayed very busy, with @livekit releasing a Private Link feature to integrate enterprise infrastructure into cloud agents, and @pipecat_ai adding support for the cool new kid, JEV, with its new Classifier capabilities.
Have a great week!
Another post about GPTLive. Even if most of the parts and WebRTC integration had been shared before I liked the prefill and handoff sections in this post.
Voice and Video AI Weekly Update! 👇
In the products side @Apple launches Siri AI, rebuilt on Apple Foundation Models with Google’s Gemini 🤯 and @OpenAI added experimental voice conversations support to Codex.
The speech-to-speech race got a new front this week: @Google's Gemini 3.8 Live can reason and call tools while it keeps talking, and @StepFun_ai's StepAudio 3 Realtime is doing the same think-while-speaking trick at the top of the Full-Duplex Bench. Full-Duplex models 🔥🔥🔥
On the STT side it was a busy week too, with @xai's Grok Voice Transcribe 2.0 doubling its accuracy at the same price and @Speechmatics launching Agent STT on its new Linden model, tuned for the digits and negations that actually break voice agents.
Consolidation continues in Europe, where @OVHcloud picked up Gladia to pitch sovereign voice AI 🇪🇺, while @nuance_ai raised $50M to build a single full-duplex model that reads gaze and gesture, not just words.
Closer to home, @tavus shipped Memories so your PALs remember people across conversations instead of starting from zero every time. And a nice reminder from @AssemblyAI and @AgoraIO that latency is a whole-pipeline problem: fast STT alone doesn't make an agent feel fast.
Have a nice week! Full update in the link below 👇












