PhD student at UVA Interested in on-device AI
Joined June 2026
- Tweets6
- Following6
- Followers45
- Likes8
Open-weight TTS models can look remarkably close to the best closed-weight models on benchmarks and demos. But are they reliable enough to actually replace them?
We stress-test Qwen3-TTS and VoxCPM2 in real-world scenarios, uncover reliability failures that standard evaluations miss, and develop an inference harness that eliminates the reliability gap. 🧵
Worst-of-10 WER on 1500+ word prompts, baseline → with LACI:
Qwen3-TTS-0.6B: 35.2% → 3.4%
Qwen3-TTS-1.7B: 47.8% → 4.2%
VoxCPM2, 1000–1300 words: 16.3% → 8.1%
The long-form gap is gone for Qwen3-TTS, and the same detector transfers across model sizes and families.
Voice cloning breaks the same way. More reference audio should mean a closer match to the speaker. For the baseline it means more failures instead: at 120s of reference, 26% of generations are catastrophic (WER above 30%) and worst-of-10 wSIM collapses to 0.01. With LACI: under 1% and 0.47, and longer references finally pay off.
wSIM is our sliding-window speaker similarity (8s windows, 4s stride). Whole-clip SIM averages short identity glitches away; wSIM exposes them.
Paper: arxiv.org/abs/2609.16989
Code will be open-sourced with publication.
