🚀 We’re excited to introduce our Speculative Decoding integration for our TensorRT-LLM Engine Builder! Our new integration allows engineers to: • Leverage SpecDec as part of our Engine Builder flow • Hit the ground running with pre-optimized configs • Lift the hood and tune for any custom use case 👉 Check out the launch blog to learn more: baseten.co/blog/speculative-… Huge shoutout to our model performance team for this feature, including @aqaderb, Justin Yi, Bryce Dubayah, and @defpan!

After witnessing so many AI builders encounter the limitations of real-time inference, we’re excited to announce async inference on Baseten! 🔄 🎉 Check out our blog to learn more: buff.ly/4cGrDr1 And don't miss our live webinar on August 15th! buff.ly/3zEyLpn

Thank you to the ML engineers, scientists, and founders who attended our NYC AI breakfast yesterday morning. New York's community of AI leaders is just getting started!

We’re excited to launch Meta’s Llama 3 in our model library, in both 8B and 70B 🎉 The newly introduced Llama 3 is a significant improvement over Llama 2, with increased tokens, and reduced false refusal rates. These models deliver unparalleled performance, showcasing significant advancements in efficiency and speed. Our Llama 3 8B runs on A100s and Llama 3 70B runs on H100s, optimized for production.