Some people are interpreting this as a way for OSS models to bypass encryption of closed model APIs. If that is what you are taking away from this, please understand that the title is a massive overstatement in this context. The point of encryption here is to keep the inference protocol stateless; I assume labs are fully aware of some side channel implications and do not care about preventing replay (why would they?) Encrypted traces offer a stateless inference protocol so inference backend can be distributed across hundreds or thousands of data centers without the need for coordinating sessions. If your side channel is to ask the LLM "true or false" questions for the universe of words that might be contained in a trace, you'd go broke decoding a single trace. So you'd have to assume there's some super special magic trace that lived within some encrypted blob that's worth attacking in this way. Overall definitely an interesting limitation of the encrypted vs stored traces, but you aren't going to be able to steal traces this way to train a competing model. That would cost more than building the model with the standard training arsenal.

Excited to welcome @tri_dao to the Together team! FlashAttention-2 will improve all LLMs everywhere in like <1 week. It’s absolutely brilliant!

The era of sub-quadratic LLMs is about to begin. At @togethercompute we've been building next gen models with large space state architectures and training them on very long sequences and the results from the recent builds are... incredible. Will share more as we get closer to releases!

Share and enjoy!

RedPajama-7B performs better at 440B tokens than all the best models trained on Pile, and continues to get better. More information on experiment design in the blog post and will keep you all posted as this converges further!

Llamas in Red Pajamas will be everywhere soon! In the meantime here’s the inimitable Rae Sremmurd for your weekend listening 😁🦙youtu.be/HqiqVZ8PJ2Q @togethercompute