I like food, and some other things too. Making ML magic 🪄 at @huggingface.
Seattle
Joined March 2023
- Tweets43
- Following75
- Followers107
- Likes29
Ann Huang retweeted
Observers: A Lightweight SDK for AI Observability
TLDR;
- Track and record interactions with AI models
- Store observations in multiple backends @huggingface, @argilla_io @duckdb
- Query and analyse your AI interactions with ease
GitHub:
github.com/cfahlgren1/observ…
We're turning @huggingface Hub's files into content-defined chunks to speed up your workflows!⚡️
This means:
- 🧠We store your file as deduplicated chunks
- ⏩ You only upload changed chunks when iterating!
- 🚀 Pulling changes? Only download changed chunks!
In our benchmarks, we found that using CDC to store iterative model and dataset version led to transfer speedups of ~2x - we'd love to learn about more real world examples to see how we perform!
We're planning roll out our new storage backend on the Hub in early 2025. Read more about CDC and our learnings! huggingface.co/blog/from-fil…
Ann Huang retweeted
Super excited to introduce Halo: A beginner's guide to DIY health tracking with wearables! 🤗✨
Using an $11 smart ring, I'll show you how to build your own private health monitoring app. From basic metrics to advanced features like:
- Activity tracking
- Heart rate monitoring
- Sleep analysis
- and more!
Ann Huang retweeted
If you were to fix one thing on the Hugging Face Hub, what would it be?
What's your biggest gripe with the hub - let's fix it; the more, the merrier.
Ann Huang retweeted
The @huggingface SQL Console now has Embeds!
🔗 Nice URL to Share / Save your Query Results
🖼️ Embed Results into Web Pages via IFrame
In this example, I use handy DuckDB regex functions to find the Code Feedback conversations with the most markdown code blocks
Ann Huang retweeted
How do you release an impactful dataset on the @huggingface Hub?
We're enhancing how we track dataset downloads on the Hub, so I wanted to share some common themes I've noticed for datasets with high downloads. 🧵
Ann Huang retweeted
Glad to see the @OpenSourceOrg release their OSAI definition process after an extensive collaborative process, and especially happy to see the role of training data enshrined!
Head over to the OSI HF org page if you want to discuss the definition on @huggingface 🤗
1/2🧵
ALT A screenshot of the Open Source Initiative organization page on Hugging Face with a link to community discussions outlined
Ann Huang retweeted
🆕 𝚜𝚊𝚏𝚎𝚑𝚝𝚝𝚙𝚡: a new open-source library from the Gradio team 🆕
This library is a product of our collaboration with @TrailOfBits and allows you to make asynchronous GET requests while avoiding Server Side Request Forgery.
A 🧵 on why this is important!
Ann Huang retweeted
The source code for HFChat macOS🤗is now fully open source and accepting PRs! Looking forward to see what folks will build.
You'll also find some hidden features that never made it to the release: github.com/huggingface/chat-…
Ann Huang retweeted
Did you know that you can load the newest checkpoints (like Llama 3.2) into Keras directly from the original HuggingFace release (safetensors)?
I tried - and lived to tell the tale: huggingface.co/blog/keras-ll…
Ann Huang retweeted
What a great day for Open Science! @AIatMeta released models, datasets, and code for many of its research artefacts! 🔥
> Meta Segment Anything Model 2.1: An updated checkpoint with improved results on visually similar objects, small objects and occlusion handling. A new developer suite will be added to make it easier for developers to build with SAM 2.
Model checkpoints: huggingface.co/collections/r…
> Layer Skip: Inference code and fine-tuned checkpoints demonstrating a new method for enhancing LLM performance.
Model checkpoints: huggingface.co/collections/f…
> SALSA: New code enables researchers to benchmark AI-based attacks to validate security for post-quantum cryptography.
Repo: github.com/facebookresearch/…
> Meta Lingua: A lightweight and self-contained codebase designed to train language models at scale.
Repo: github.com/facebookresearch/…
> Meta Open Materials: New open source models and the largest dataset to accelerate AI-driven discovery of new inorganic materials.
Model checkpoints: huggingface.co/fairchem/OMAT…
> MEXMA: A new research paper and code for our novel pre-trained cross-lingual sentence encoder covering 80 languages.
Model checkpoint: huggingface.co/facebook/MEXM…
> Self-Taught Evaluator: a new method for generating synthetic preference data to train reward models without relying on human annotations.
Model checkpoint: huggingface.co/facebook/Self…
> Meta Spirit LM: An open-source language model for seamless speech and text integration.
Repo: github.com/facebookresearch/…
Sweet visualization of S3 PUT requests to @HuggingFace Hub over a 24-hour period, showing upload density across the globe. 🌏
Ann Huang retweeted
New blog post: Scale AI-based Data Processing EASY
The FineWeb-Edu dataset comes from processing 45TB (🤯) of FineWeb
And it uses a Language Model to classify the educational level of the text 😭😭
Still, we reproduced it in a few lines of code !
The key ? HF + Dask 😎
Did you know that @Huggingface Hub holds over 29 PB of Git LFS files across datasets, models, and spaces? 📈
That's the equivalent of 64 @CommonCrawl downloads - and it's growing every day. So what's inside? 🧵
As a first step to optimizing storage, we checked to see how many of the files we stored were exact duplicates: 3.24 PB! That means that if we just implemented file-level dedupe across all repos, we'd save 10% of the storage space.
With that change, forks of @HuggingFace repos would all be essentially free storage-wise! Then to tackle block-level dedupe...
Check out the full space: huggingface.co/spaces/xet-te…