𝔖𝔢𝔥𝔯𝔦𝔰𝔥 𝔨𝔥𝔞𝔫*

Drop 2/15 Are we actually measuring the things we ultimately want our voice AI models to achieve? The best AI voices are becoming hard to tell apart. From each other. Not from humans. 🚀 Introducing Turing TTS: an evaluation that measures how often human listeners mistake AI-generated speech for real human speech. We tested leading TTS models against real human speech. Even the best model fooled listeners only ~10% of the time. With Turing TTS, we hope to contribute to a broader discussion within the research community about how we measure progress in voice AI. If one of our goals is to develop speech that is indistinguishable from human speech, how well do our current evaluation methods capture progress toward that goal? We don’t claim that Turing TTS is the right way forward, or that human indistinguishability should be the goal of every TTS system. Rather, we hope these findings encourage the community to collectively examine whether blind pairwise human evaluations of naturalness are sufficient, and what else we might need to measure. Results are now live on @voicearena_ai.