Common Crawl Foundation retweeted
In 1950, five people at MIT passed notes through slots. Today the math from that experiment helps decide what AI gets trained on.
stephenburns.ai/blog/harmoni…
Common Crawl Foundation retweeted
Replying to @CommonCrawl
My bet: low-resource languages would lose at the filter.
Quality classifiers trained mostly on English throw out good Icelandic pages as noise. Smarter crawl budgets won't help if the filter can't read the language.
we will have two papers @NeurIPSConf this year from the croissant community 🥐
1. "croissant baker" led by rafi al attrach and @tompollard
2. "croissant miner" led by berke arda and @akhtarmubashara
i took a quick glance at the rearview mirror
since early 2023, our idea for a unified ml data access layer has grown into an expansive ecosystem, incl
- data aggregators spanning hundreds of thousands of datasets incl @huggingface @kaggle @dataverseorg @CommonCrawl and others
- tooling spanning auto-generation and -verification of croissant files
- the formal spec maintenance under @MLCommons
- and a growing list of community extensions spanning verticals such geo, responsible AI, health and others
onwards towards the composable data dream(:
Common Crawl Foundation retweeted
There’s an enormous amount of open training data on Hugging Face. What does it take to make it work together 🤗?
For Marin’s 535B run, we built on 25T tokens from 152 datasets with licenses permitting training.
Here’s the work between downloading those and training a model 🧵

ALT Flow diagram of the Marin data pipeline. Starting with 25.25 trillion tokens, deduplication removes 2.13 trillion and benchmark decontamination removes another 13.66 billion, leaving 23.11 trillion. Documents are organized into 40 topics and five quality bands, yielding 200 sampling groups. Results from 872 small-model experiments feed a regression used to choose sampling weights.
the modern web you see incl websites you browse will likely disappear or become irrelevant altogether in the next few years. they’ll feel like absolute ghost towns.
when the previous era existed (pre ai) google aggregated attention & then sprayed traffic to the web based on intent.
modern ai has no need to spread traffic like this, it just needs to do something through an agent to agent negotiation. this leaves websites & web interfaces completely irrelevant.. which renders the browser pretty obsolete over time, just go look at your browser history & see how top heavy it all is.
this is why ai is such a giant game to win because it is the ultimate aggregation of attention / time spent. you never even have to leave like you did google! whoever controls this basically controls a large portion of the gdp. e.g. there might be a measure of success metric that is something like gdp per ai agent.
Common Crawl Foundation retweeted
Excited to share a collaboration between @QatarComputing and @CommonCrawl to strengthen Arabic LLMs.
We contributed 30K+ Arabic domains, enriched with quality annotations and metadata, to improve Arabic coverage in future Common Crawl releases.
🔗 commoncrawl.org/blog/turning…
Common Crawl Foundation retweeted
Have you ever seen a user agent named "CCBOT"?
If so, you were visited by @CommonCrawl, a non-profit that crawls the internet and publishes a 10+ petabytes open-source dataset.
I think it's beautiful that humanity shares this data.
It means that anyone with minimal resources has the access to data required to build their own AI models.
It also means we don't have to crawl the entire internet thousands of times for each research, saving large amounts of bandwidth and resources.
Common Crawl Foundation retweeted
Mistral CEO: AI companies should pay a content levy in Europe ft.trib.al/hKU8k0g | opinion
Common Crawl Foundation retweeted
I've been dabbling with Claude Code. Using English as a programming language, I wrote a C compiler. medium.com/@kazabyte/english…
Common Crawl Foundation retweeted
Announcing our latest paper: CommonLID: Re-evaluating State-of-the-Art Language Identification Performance on Web Data
In collaboration with @CommonCrawl @MLCommons and @JohnsHopkins we worked with 80+ native speaker annotators to build a LID benchmark on actual Common Crawl text covering 109 languages. Existing evaluations overestimate how well LangID works on web data.
Common Crawl Foundation retweeted
The AI backlash is here.
This post has a million views.
People hate technology.
They hate innovation.
They see that it is coming for their jobs.
Coming for their way of life.
Innovation is asking them to do more work. To spend nights and weekends learning about AI and building new things.
They hate that. And I've seen it my entire career as people tell me over and over again that they don't like what I'm pitching them.
How many have gone to an Apple store and had a demo of the Vision Pro? Even in Silicon Valley I keep finding smart people who refuse. And my consumer research outside the valley shows far far greater resistance.
There are couple other such posts this morning going viral:
@zachweinberg: xitter.cf/zachweinberg/status/20…
@tszzl: xitter.cf/tszzl/status/200248841…
Here's the real problem.
Let's say you burn down Silicon Valley.
The Chinese are coming and coming with cars that will eviscerate the rest of the auto industry. Americans can't see that because we tax the heck out of them and keep them from selling here.
And soon they are coming with robots. Americans can't see that because they aren't like me meeting with robot pioneers here and in China. Or watching the robot community here on X (or understand what they are talking about with their highly technical talks about world models and reinforcement learning.
And they are coming with AI models. Most Americans are sticking their heads in the sand and ignoring what's going on on @huggingface and here on X. I talk to entrepreneurs around the world who keep telling me they are building businesses on Chinese models because they are cheaper and better.
And they are coming with entertainment businesses that will eviscerate Hollywood. Most Americans see AI artists as anti human. The Chinese don't.
The only way out of this +is+ innovation. That's where any new jobs are gonna be.
So go ahead and burn down Silicon Valley and San Francisco.
That won't save your job and will doom America to being a poor country subservient to the Chinese.
And, yes, we need to do a better job of explaining the role of innovation.
But that job is bigger than what I can do.
Why?
Because people hate innovation and want to stick their heads in the sand, and even, roll back innovation, my posts aren't getting any reach here on X outside of a small group of innovation supporters.
Well, I'll keep supporting the innovators. I made lists of almost all of them, 37,000 in AI alone, and 7,600 AI companies: xitter.cf/scobleizer/lists
But that isn't nearly enough. The entire industry needs to do a better job of talking to everyday people and getting them to see a better world is possible.
For now all they hear is "you are taking my job."
And they aren't wrong about that either. Which makes this all so difficult to get people onto the side of innovation.
Common Crawl Foundation retweeted
"You shouldn't have put your content on the internet if you didn't want it to be on the internet."
— Common Crawl's executive director Rich Skrenta
Common Crawl Foundation retweeted
“When only a few have the resources to build and benefit from AI, we leave the rest of the world waiting at the door,” said @StanfordHAI Senior Fellow @YejinChoinka during her address to the @UN Security Council. Read her full speech here: hai.stanford.edu/policy/yeji…
Common Crawl Foundation retweeted
My latest article "Train-Set SEO: Why Embedding Your Brand in AI’s DNA is the Future of Search Optimization" just got published on @hackernoon 💚💚
hackernoon.com/train-set-seo…
Common Crawl Foundation retweeted
We don’t give nearly enough credit to the people and organizations who build and share open AI datasets.
In fact, I’d argue they matter even more than open models:
- they’re foundational, enabling hundreds of different models
- they remove not only technical but also legal bottlenecks for democratization of AI building
- their impact usually lasts longer, since datasets are less tied to the latest training fad.
Let’s celebrate the teams who are sharing open datasets today. Who are your favorite ones?
Common Crawl Foundation retweeted
One of the biggest obstacles to improving language technologies for low-resource languages is the lack of data. To address this, we need better language identification tools. So, we're organizing a shared task on Language Identification for Web Data!
Common Crawl Foundation retweeted
"The most valuable resource isn't data, it's the ability to transform data from an abundant commodity into verified intelligence"
- Ben Jorgensen, Constellation CEO
Constellation’s new product, Digital Evidence, launched @DigitalChamber DC Summit!
🌐constellationnetwork.io/digi…
Common Crawl Foundation retweeted
Excited to launch our partnership with @CommonCrawl to enhance tools and datasets for AI researchers
First up, the Common Crawl Agent: commoncrawl.org/ai-agent
ReadyAI’s structured data pipeline turns thousands of records into detailed insights to get you started training AI today
The single largest source of data for GPT, Llama and EVERY SOTA model is @CommonCrawl
Couldn't be more excited to announce @ReadyAI_ has partnered w/@CommonCrawl to enhance tools and datasets for AI researchers. Our first collab...the Common Crawl Agent.
Try it here 👇
commoncrawl.org/ai-agent
ReadyAI’s structured data pipeline converts thousands of records into metadata enhanced vector embeddings for extremely accurate insights, helping AI researchers and developers quickly build, test, and deploy smarter models.
More details on the partnership here: readyai.ai/p/readyai_x_commo…
Common Crawl Foundation retweeted
I’ll be today at the AI Action Summit in Paris, if you’re attending and want to discuss about @CommonCrawl or about open data, please DM me!































