🦙 Llama 4 is here! → Llama 4 introduces three models: Scout (17B active parameters/16 experts), Maverick (17B active parameters/128 experts), and Behemoth (288B active parameters/16 experts), with only Scout and Maverick being released now. → These are Meta's first natively multimodal models and first to use mixture-of-experts (MoE) architecture. They claim they outperform much larger competitors like GPT-4o on various benchmarks. → Llama 4 Scout offers 10 million token context window (!) thanks to a new "iRoPE" architecture, dramatically improving from Llama 3's 128K context. → Training involved novel techniques including "MetaP" for hyper-parameter selection, continuous online RL with adaptive filtering, and codistillation from the larger Behemoth model. → Meta claims significant progress on reducing political bias, with Llama 4 refusing less on debated topics (down from 7% to below 2%) and showing more balanced responses. It's cool to see the technical innovations Meta has packed into Llama 4, particularly the massive context window and the MoE architecture. The multimodal capabilities put them in direct competition with GPT-4o and Gemini models.

Llama 4 is here with 2T, 400B, and 109B MoEs. This bad boy just got outllamaed. ai.meta.com/blog/llama-4-mul…