LLM speed, throughput, prefill, generation, TTFT, 😕? Inside Groq we've standardized on the following terms to reduce the confusion: 1. Rate - the user experience, the rate at which a single user's tokens are input or output. 2. Throughput - the economics, how many tokens the HW is capable of producing across all users. 3. Input - input is tokens that are sent into the LLM. 4. Output - output are the tokens that the LLM, well, outputs... 5. Round-Trip - the total amount of time added to get to / from the user. This is variable and depends on where the user is located, and what their Internet / WiFi situation is. What users care about is input rate and output rate, since those two metrics determine their experience. What those running the service care about is input throughput and output throughput, since that determines their costs. Note that these numbers are not independent, typically the faster the rate the lower the throughput, GPUs have to trade this off, this isn't the case with LPUs though - the fast the rate on an LPU typically the faster the throughput. What about TTFT, or Time To First Token (the term also confusingly and previously known as prefill...) TTFT depends on the size of the input, e.g. if your input rate is 30,000T/s, then it would take 10ms to input a 3,000T input, and 20ms to input a 6,000T input. TTFT is therefore dependant on input size, and thus not an ideal measure, even though it is what a user cares about. What matters here is the Input Rate and the Round-Trip time, those two will tell you what a particular user experiences. This is the terminology we use inside of Groq, and isn't standard - because there is no standard. While we don't expect others to adopt these terms, they've reduced the confusion considerably inside of Groq so we decided to share them. Now go back to enjoying your, hopefully slightly less confused, Monday 😁

Please no one name any models grizzly or honey badger.