ontocord retweeted
[1/5]
💥 Facing the LLM Scaling Challenge Head-On! 💥
Glad to introduce MGA: Reformulation for Pretraining Data Augmentation!
The AI world is grappling with data limitations and the performance hit from data repetition.
We introduce MGA (Massive Genre-Audience) Reformulation – a groundbreaking, scalable data augmentation method to revolutionize how we train LLMs!
📄 Paper: arxiv.org/abs/2502.04235
#LLM #DataAugmentation #AIResearch #Innovation
ontocord retweeted
Releasing another fastText classifier with @ontocord - this time is to filter math from web data.
@armandjoulin @EXGRV @p_bojanowski Tomas Mikolov's FastText ages very well, one of the secret behind pretraining data curation as its throughput is 2000 examples per second in CPU!
Imo, FastText is the best practice. It is quick to train and scales well. If you look at the technical reports, @deepseek_ai and @Alibaba_Qwen team used them to curate their maths model (they didn't open source model though).
The whole collection has over 300,000+ downloads:
Maths: huggingface.co/kenhktsui/mat…
Codes: huggingface.co/kenhktsui/cod…
Educational Value: huggingface.co/kenhktsui/llm…
huggingface.co/kenhktsui/fin…
What do you think—fastText still the GOAT for scalable data curation?
ontocord retweeted
LAION presents BUD-E & School-BUD-E,
100% Open Source browser-based Voice Assistants, that work out-of-the-box with self-hosted APIs, Google AI Studio, Groq, SambaNova, Fish TTS, Deepgram, MARS 6 & Hyprlab, saving the user data locally in the browser. 😃
laion.ai/blog/bud-e-release/
ontocord retweeted
Grass, Ontocord, and LAION launch VALID
We’re excited to share the release of VALID (Video-Audio Large Interleaved Dataset) by the world-renowned teams at @ontocord and @laion_ai. VALID was built using the Grass Video Repository.
This dataset is comprised of 30 million audio snippets interleaved with images and text, making it the FIRST EVER video-audio interleaved dataset.
ontocord retweeted
LAION-5B is important reference research dataset for reproducible language-vision foundation models studies. We release Re-LAION-5B as a transparent safety iteration on LAION-5B which fixes issues and allows broad research community to continue using open datasets as reference🧵
huggingface.co/blog/kenhktsu…
Fineweb-edu comparable classifier at blazing speed - 2000 docs/sec in CPU!
arxiv.org/abs/2404.08676
🚨Introducing ALERT🚨 , a large-scale safety benchmark of > 45k instructions categorized using our new taxonomy.
✨Using prior work such as Bo Decoding Trust and our Aurora-m huggingface.co/datasets/auro…
and @jackclarkSF's awesome Anthropic-HH.
🙏
ontocord retweeted
For more details, check out our preprint: arxiv.org/pdf/2404.08676.pdf 🤓
Huge thanks to @felix_friedri, Patrick, @kerstingAIML, @RNavigli, Huu and Bo (and all the organizations involved @babelscape @SapienzaNLP @TUDarmstadt @Hessian_AI @DFKI @ontocord @UChicago @uiuc_nlp)🫂
ontocord retweeted
Aurora-M
The First Open Source Multilingual Language Model Red-teamed according to the U.S. Executive Order
Pretrained language models underpin several AI applications, but their high computational cost for training limits accessibility. Initiatives such as BLOOM and
Announcing our print - an Ontocord.AI open science 16b model to prompte lawful AI: **Aurora-M** red-teamed for concerns under #WhiteHouse Executive Order on the Safe, Secure, and Trustworthy development and use of AI - arxiv.org/abs/2404.00399 So proud of our team!
Announcing our official launch of the Aurora-M series of multilingual models red teamed for the Biden-Harris AI Executive Order concerns.
Blog: huggingface.co/blog/mayank-m…
Thanks to the MDEL community & to CSC - IT Center for Science and TurkuNLP for compute.
ontocord retweeted
@TeraflopAI is excited to help support the @caselawaccess and @HarvardLIL, in the release of over 6.6 million state and federal court decisions published throughout U.S. history.
ontocord retweeted
Today, we are announcing Occiglot!
A large-scale collaborative research collective focusing on open-source European LLMs.
We invite anybody working on multilingual datasets, benchmarks, or models to get in touch/join our discord.
occiglot.github.io/occiglot/…
ontocord retweeted
🚀🥯 bagel 20b v0.4 family now available 🥯🚀
Fine-tunes of internlm2-20b, with the latest bagel dataset.
Prompting tips and such in the model card.
4 options to choose from:
• DPO with internlm2 modeling code huggingface.co/jondurbin/bag…
• non-DPO with internlm2 modeling code huggingface.co/jondurbin/bag…
• llamafied DPO with internlm2 modeling code (required for vllm) huggingface.co/jondurbin/bag…
• llamafied non-DPO with internlm2 modeling code (required for vllm) huggingface.co/jondurbin/bag…
Compute for the SFT phase of the model was provided by massedcompute.com, and the DPO phase was sponsored by @latitudesh, using the new launchpad system!
