Bot info
What this crawler is
AI Readiness Check is a tool that analyzes a news website's AI readiness when someone submits its URL. When a URL is submitted, our scanner fetches that site's publicly accessible pages to evaluate crawler access, licensing and rights signals, attribution and structured-data markup, and whether AI-powered search tools actually cite the site's journalism. It runs only on demand — it does not crawl the web continuously.
How we identify ourselves
When our scanner fetches a site's pages, it sends this User-Agent string:
Mozilla/5.0 (compatible; AIReadinessCheck/1.0; +https://crawlready.org/bot-info)
The AIReadinessCheck token is the name you'd use in a robots.txt rule aimed at us.
How to block us
To stop AI Readiness Check from analyzing your site, add this to your robots.txt:
User-agent: AIReadinessCheck Disallow: /
To block us only on a specific path, use that path instead of / (e.g. Disallow: /private/).
Do we respect robots.txt?
Yes. Before we analyze a submitted site, we fetch its robots.txt and check whether our user-agent is allowed to access the URL that was submitted. If your robots.txt disallows AIReadinessCheck (or a * rule that covers that URL), we do not fetch any of your pages and we do not generate a report — we return a short notice saying we were disallowed.
We check the robots.txt rule against the submitted URL's path. A site-wide Disallow: / blocks us entirely. During an analysis that we're allowed to run, we also fetch related public pages (articles, policy pages) and do not re-check robots.txt for each individual sub-page — so a path-specific block is honored for the URL you submit, but a path you didn't submit that's separately disallowed wouldn't stop a sub-page fetch inside an otherwise-allowed run.
AI-monetization gating probe
For one specific check — detecting whether a site has deployed AI-monetization gating infrastructure (TollBit, Cloudflare Pay Per Crawl, or similar x402-based systems) — our scanner sends two read-only requests: one identifying itself as GPTBot and one as ClaudeBot, rather than as AIReadinessCheck. This is necessary because that infrastructure gates based on the requesting crawler's identity, and some sites gate selectively — allowing one AI company's crawler while billing another. Probing under both identities is the only way to detect that pattern.
We do this only when your robots.txt allows the specific crawler. If your robots.txt disallows GPTBot, we skip the GPTBot probe; if it disallows ClaudeBot, we skip the ClaudeBot probe. We do not probe under any identity you've asked not to visit.
No payment credentials are ever sent. No quoted price is ever accepted. Each probe records only the HTTP status code and the names of payment-protocol headers — never their values. This is the only check that uses identities other than our own.