We’re excited to announce a new update to OpenRobotsTxt.org!
You can now search the archive for individual websites’ robots.txt files and look up specific bot names.
blog.majestic.com/company/ne…
OpenRobotsTXT retweeted
Interested in user-agent data? @majestic has you covered -> Majestic Launches Robots.txt Archive
600M hostnames scanned and 36K user-agents found.
"The project has been bootstrapped by a huge data export of robots.txt files collected by the Majestic crawler, MJ12bot. This has enabled us to analyze the User Agents reported around the web. The initial release of the site focuses on this study with a free-to-download (Creative Commons) data set that details the User Agents discovered across the web." blog.majestic.com/company/ma…
OpenRobotsTXT retweeted
Cool new project from @Majestic 👀
We're delighted to announce OpenRobotsTxt.org - a project to archive and analyse the world’s robots.txt files, kicking off with analysis of 595 million hostnames courtesy of a huge export of data from @Majestic.
So far we have seen almost 37,000 User-Agents mentioned in files on more than 600 million hostnames!
openrobotstxt.org
We're delighted to announce OpenRobotsTxt.org - a project to archive and analyse the world’s robots.txt files, kicking off with analysis of 595 million hostnames courtesy of a huge export of data from @Majestic.






