Interested in user-agent data? @majestic has you covered -> Majestic Launches Robots.txt Archive
600M hostnames scanned and 36K user-agents found.
"The project has been bootstrapped by a huge data export of robots.txt files collected by the Majestic crawler, MJ12bot. This has enabled us to analyze the User Agents reported around the web. The initial release of the site focuses on this study with a free-to-download (Creative Commons) data set that details the User Agents discovered across the web." blog.majestic.com/company/ma…
We're delighted to announce OpenRobotsTxt.org - a project to archive and analyse the world’s robots.txt files, kicking off with analysis of 595 million hostnames courtesy of a huge export of data from @Majestic.
ALT The words "OpenRobotsTXT" on a teal background.