🪩 Introducing Llama3-German-8B! A large language model specialized for German, built by @DiscoResearchAI and @occiglot. Based on @Meta's Llama3-8B, it's trained on 65B high-quality German tokens. Available now at huggingface.co/DiscoResearch…! 🕺

Come visit our discord to chat with us about LLMs and share the cool models you build on this release: discord.gg/V7Bm6Vu44S! And visit ellamind.com to find out more about how we can help you bring customized LLMs into your business 🚀

Many thanks to our friends at @occiglot, especially @MBrack_AIML for this fun collaboration! Also thankful for @laion_ai and friends for initiating this project and collaboration.

Model training was supported by a compute grant at the 42 supercomputer at @Hessian_AI, the AI Innovation Lab (funded by the @HMWK_Hessen & the HMinD) and the AI Service Centers (funded by the @BMWK).

We reproduced the results from the "Fewer Truncations Improve Language Modeling" paper by Ding et al. (arxiv.org/abs/2404.10830v2) using an FFD bin packing. Our experiments show improvements of up to 3.32% vs equi-data naive packing, confirming the effectiveness of this approach 🔥

We're also releasing a long-context version trained on another 100m tokens of 32k sequence length data with an adjusted rope_theta of 1.5e6 to fulfill all your RAG and summarization desires! huggingface.co/DiscoResearch…

We trained on the German split of @occiglot’s high-quality multi-lingual occiglot-fineweb-dataset. The dataset comprises multiple diverse data sources, and is meticulously cleaned and deduplicated. Further details are available here: occiglot.eu/posts/occiglot-f…

🔎Training details: - 128 A100s on @Hessian_AI's 42 supercomputer for 60h - 15,500 steps with a cosine learning rate schedule, 1e5 LR and 1% warmup - Batch size of 4M tokens Trained and evaluated by @bjoern_pl. More details in the model card 🔎

🚀Key features of Llama3-German-8B: - Improved German performance with minimal English degradation - Long-context version capable of processing 32k tokens - Instruction-tuned variant using DiscoLM German dataset - Intelligent document packing

Llama 3 🦙🦙🦙 enters the dance floor 💃🪩🕺 DiscoResearch presents its very first (very) experimental fine-tune: Llama 3 DiscoLM 8b German v0.1 Experimental Not production ready, but ready to dance! 😉 Model: huggingface.co/DiscoResearch… Demo: 364b61f772fa7baacb.gradio.li…

Checkout #Occiglot a large-scale research collective for open-source development of LLMs: huggingface.co/occiglot A great open model initative by many of our friends @Hessian_AI , @DFKI, @TUDarmstadt and we are happy to contribute a (small) dataset for instruction tuning 🙂.

Repo/code, more benchmark results and bigger models will follow - Join the #discolm_german channel on our Discord for news, feedback + feature requests: discord.gg/UPVUkT9rnQ 👋. As usual many thanks to @Hessian_AI + @laion_ai for their support 🙏🙂.

Download: huggingface.co/DiscoResearch… Demo: demo.discoresearch.org/ The model has been trained in 3 phases (LeoLM ctd pretraining -> SFT -> DPO) and offers a specialized RAG template (by @rasdani_) and (experimental) function calling ability. 2/3

We have launched DiscoLM German 7b 🥳.This model delivers exceptional German language quality and is able to switch seamlessly between German and English;just prompt in the preferred language! Goal was to create a model that can be a drop-in-replacement for GPT-3.5;DL+Demo 1/3👇

Hard question 😜

DiscoLM Mixtral 8x7b is now available for inference with up to 100 tokens per second at @togethercompute. Blazingly fast release; blazingly fast inference. 🙌 Try it out yourselves at together.ai/blog/mixtral! 🪩