honestly speaking most times one is enough. 2 is just a safe guard I have since prefix caching lives on one servers GPU's anyway. the only thing is that if it lands on replica 1 and the first turn routes to replica 2 it doesn't help.
so when I was benchmarking this and testing just kept 2. cost is minimal anyways but yes you can easily just do 1! good q
yup! good cascaded voice agents is just all these optimizations to try to replicate the behaviour of s2s lol
honestly i think it still depends, but right now s2s is much less controllable for developers, especially for custom voices. Most s2s systems support cloning, but it鈥檚 harder to synthesize well bc you鈥檙e going audio-to-audio instead of using a dedicated TTS model.
The model has to disentangle content, prosody, and speaker identity from the source, which creates trade-offs in timbre (vocal diff) leakage and prosody that a cascaded TTS pipeline avoids. so it has its pros and cons
ai engineers ship great models and awful first impressions.
cache warming is the easiest, highest-leverage way to cut latency, yet almost everyone ignores it, even in voice ai. that obsession with the first second is one reason why @boardyai stands out!
completely agree, literally wrote about this yesterday! prefill + prewarming cache.
the text-output problem I mention above is more so about the paradigm shift of using an audio <> action model over a LLM intermediary for reasoning
devp.ca/synthetic-warmup
I do think that voice ai pipelines still route almost everything through full ASR to generative decoding. A lot of latency and cost sits in that text-output step so reformulating the problem as constrained classification (audio to action) instead of open generation is def going to be the new paradigm shift
