Scientist professor hacker writer citizen. Committed to building a better world through science technology and community. Director @swheritage. Follow ≠ endorse
Paris
Joined June 2009
- Tweets5.5K
- Following5.4K
- Followers5.1K
- Likes3.4K
Pinned Tweet
Yes, source code is special: it's executable knowledge that is meant to be human readable, by design. It's an all time new in human history.
The AI crawler mess had a solution 30 years ago, never applied. A textbook market failure. N sites, M crawlers, and a bill that lands on nobody's balance sheet. 👉 dicosmo.org/good-enough/ @inria @cnrs @GoogleDeepMind @OpenAI @Microsoft @huggingface @AnthropicAI @unesco @ODET_UN
Crawling the whole web to find out what changed was never sensible. It was patched until it was bearable, and then it became normal, like a QWERTY keyboard. That is what "good enough" means, and it is why the waste lasted thirty years. 👉 dicosmo.org/good-enough/
Today's AI crawlers re-fetch what never changed. In 1998 I co-wrote the fix and sent it to the IETF; we revised it in 1999 and never heard back. In 2001 an IETF group reinvented it independently — theirs expired too. Neither was ever rejected. 👉 dicosmo.org/good-enough/
Roberto Di Cosmo retweeted
The destination, from the outset: I direct Software Heritage, a non-profit organisation that collects public source code once so that others need not collect it again.
This is my latest effort at a long-standing problem. Follow the full saga on 👉
dicosmo.org/good-enough/
The destination, from the outset: I direct Software Heritage, a non-profit organisation that collects public source code once so that others need not collect it again.
This is my latest effort at a long-standing problem. Follow the full saga on 👉
dicosmo.org/good-enough/
It is time to call that out and act on it — starting by understanding it: which actor does what, who gains, who loses.
Over the coming weeks I will reconstruct one instance from dated documents, including my own past and ongoing efforts.
For nearly three decades I have met the same pattern: a system that works well enough for each actor separately, while imposing a growing cost on all of them together.
Nobody owns the aggregate problem, so the better solution stays nobody’s job.
Everybody running a sizeable website knows the method is wasteful. The striking part is that we speak as though it were a law of nature. It is not. We knew how to avoid most of this waste in the 1990s, and I tried to push a concrete solution then.
None of this is new. Search engines have done it for decades, in the very same wasteful way.
What is new is the scale — and that the open web is closing in response. An API at a time. A "bot filter" at a time.
The mechanism, from the same report: some botnets revisit an entire site every day merely to discover what changed, transferring the same material again and again.
