AI Readiness Check™: Scoring Engine & Valuation Framework
Version 2.6 · Scoring v10 · Calibration v3 · November 2, 2026
The AI Readiness Check platform evaluates publisher digital properties for AI readiness (whether a site has decided and signaled its AI stance), passive edge technical governance, and modeled economic exposure. This document outlines the scoring rubrics, econometric models, empirical data benchmarks, and inherent limitations of our automated audit engine.
Readiness: decided and signaled
The readiness level is the headline result of every report. It answers one question: has the site decided how AI may use its journalism, and said so where machines and people can find it? It does not measure whether a site blocks AI. A site that allows every AI crawler can be fully ready; a site that blocks them all can still have gaps.
Three signals, each a yes or no from what the scan observes on the public site:
- AI crawler stance: robots.txt names at least one AI crawler, uses the shared
__usage_ai-trainingtoken, or carries a Content-Signal line or an IETF Content-Usage rule or header. - Rights position: an AI policy page that addresses outside use of the site’s content, a TDM reservation, an RSL license, an ai.txt file, or a noai robots meta tag.
- Attribution markup: the sampled article carries Article or NewsArticle structured data rated strong or partial.
Levels. Strong means every assessed signal is present; Moderate means all but one; Low means two or more are missing. If no article could be sampled, attribution is not assessed and the level is judged on the other two. If the site could not be assessed or its robots.txt could not be fetched, no level is given.
Findings follow the same rule. Allowing AI training is never reported as a critical issue. A site with no declared stance gets a “No AI training stance declared” finding; a site that declares a stance and allows training gets an informational note, recorded as a choice.
AI citations. Every single-site scan, and any batch run with citation testing switched on, asks an AI search tool up to three distinct questions related to the site’s coverage and checks whether the site is cited. One question is asked twice as a consistency check, because answers vary from run to run. Results are reported as distinct questions cited (for example, “cited in 2 of 2”), with the repeat reported separately. A handful of questions is an indication, not a measurement.
The 0–100 score is still shown alongside the readiness level, unchanged, so batches stay comparable across versions. It measures crawler blocking (Protection), AI search visibility, and legal and policy signals, as described in the sections below. Each scan records the scoring version that produced it, and batches run on an earlier version are marked as such.
The Production-to-Extraction (P2E) Ratio
Original journalism operates with high fixed operating expenses, sustained legal accountability, and professional labor standards. Automated crawling systems ingest published news reporting to train model weights or generate conversational answers.
We quantify the relationship between human editorial expenditure and metered crawl acquisition through the Production-to-Extraction (P2E) Arbitrage Ratio — a multi-factor econometric model rather than a single-ratio calculation. The model synthesizes modeled editorial operating investment (newsroom labor, legal accountability, and professional production standards) against programmatic clearinghouse rates quoted per crawl event ($/hit), net of the edge egress cost attributable to serving that automated traffic.
- Interpretation: A P2E ratio of 300x to 600x indicates that a newsroom invests roughly $300 to $600 in original human reporting for every $1 of crawl value captured at prevailing clearinghouse rates.
- Numerator (Editorial Investment): Modeled with a five-tier newsroom production ladder keyed to catalog size. Each tier pairs an annual new-article count with a per-piece cost, set from regional newsroom staffing baselines — BLS wage data and NewsGuild contract minimums (see Section 5). The engine assigns the tier whose catalog-size breakpoint the site's catalog clears, then multiplies that tier's annual article count by its per-piece cost.
- Denominator (Metered Crawl Value): Modeled based on observed monthly crawler activity priced at prevailing programmatic clearing rates ($0.015–$0.1 per call), rather than the downstream enterprise valuation of AI platforms.
| Tier | Catalog size | Modeled annual investment |
|---|---|---|
| Local Digital Native (scales with catalog size) | under 20,000 articles | $112.5K |
| Small Community Daily | 20,000–100,000 articles | $600K |
| Regional Independent | 100,000–500,000 articles | $1.5M |
| Major Metro Daily | 500,000–1,500,000 articles | $5.25M |
| National Flagship | 1,500,000+ articles | $15M |
Calibration anchors. The two published newsroom staffing baselines bracket the middle of the ladder rather than its full span: the community baseline ($1.2M–$1.8M) brackets the Regional Independent tier, and the metro baseline ($4.5M–$6M) brackets the Major Metro Daily tier. The two lowest tiers sit below the published community baseline.
Microeconomic Edge Telemetry (Modeled Estimates)
Because an external scan operates outside a publisher's private web server and origin analytics, all network volumes and dollar valuations are modeled estimates derived from domain index size, publishing frequency, and passive edge headers:
Modeled Crawl Demand
Estimated monthly AI crawl volume is derived from the site's modeled organic search visits and from Cloudflare Radar's observed bot traffic for the news and media industry, instead of a fixed per-page coefficient. The engine multiplies monthly organic visits by Radar's network-wide ratio of AI-bot requests to human requests, both taken from the same industry-filtered client-type measurement, treating one visit as one human request:
AI crawl requests = monthly organic visits × (AI-bot client-type share ÷ human client-type share)That volume is then split using Radar's crawl-purpose breakdown into four reported figures: training-crawl volume (model training), mixed-purpose crawl volume (crawls that index and train at once, primarily Googlebot and Bingbot), search/citation-crawl volume (search plus on-demand user fetches), and undeclared. Mixed-purpose crawling is reported on its own line rather than folded into either neighbor, because it cannot be blocked for AI training without also leaving search — folding it in would overstate the blockable training volume or understate citation volume. Only the training figure is reduced by a site's training-crawler rules, and total AI crawl volume is the sum of all four. Catalog size no longer drives demand — it feeds the archive floor and the editorial tiers.
Blocking reduction. The defensive reduction applied to training-crawl volume is the sum of Radar's observed per-crawler crawl-volume shares for the AI training crawlers the site actually blocks, normalized to training-crawl volume — so blocking GPTBot and ClaudeBot reduces modeled training hits by their measured combined share, not by a fixed discount. Stated assumption: unranked crawlers. Radar publishes per-crawler shares for its top-ranked crawlers only and aggregates the rest of AI crawl volume into an "other" bucket. Treating a blocked crawler inside that bucket as zero would both understate its block and make a full block unreachable — a site that disallows every crawler we track could never reduce modeled training volume to zero. Each scan therefore divides the snapshot's "other" bucket evenly across every crawler in our tracked training and citation lists that Radar does not rank individually in that snapshot, and applies that allocated share to whichever of them the site blocks. It is an assumption, not a measurement: it treats each unranked crawler as worth the same as any other, and it is bounded by the bucket, so the allocation never exceeds the volume Radar did not attribute. Every report records which blocked crawlers used a published Radar share and which used the allocation. Mixed-purpose and search/citation volume are never reduced by training-crawler rules, and the crawl-to-refer ratios shown in reports are context, not an input to any figure.
Source and window. Cloudflare Radar API, over a 28-day window, refreshed weekly. Each figure comes from a named endpoint: the crawl-purpose breakdown (ai/bots/summary/crawl_purpose) supplies the four demand-split volumes; the per-crawler AI crawl-volume shares (ai/bots/summary/user_agent) supply the blocking reduction, together with the unranked "other" bucket they are drawn from; the per-platform crawl-to-refer ratios (bots/crawlers/summary/crawl_refer_ratio) are report context only. The demand ratio itself uses a single endpoint for both of its terms — bots/crawlers/summary/client_type, industry-filtered, supplies the AI-bot share and the human share — so numerator and denominator describe the same population. Radar's HTTP bot-versus-human split of HTML requests (http/summary/bot_class) ignores the industry filter; it is stored on the snapshot as context and is not divided into the ratio. The snapshot is stored with the window, the industry filter applied, both ratio inputs and the exact endpoint URLs used, and each report cites the window its figures came from.
Vendor caveat. Cloudflare sells bot-management and pay-per-crawl products, so these measurements come from a vendor of the remedy they describe. They are the best network-wide measurement of AI crawl traffic available to us, and every figure built on them is labeled as Radar's observation rather than a measurement of our own.
Clearing rates reflect commercial API price cards across programmatic clearinghouses (e.g., TollBit) with modeled bands spanning $0.015–$0.1 per call.
Legacy model. A scan that runs without a stored Radar snapshot falls back to the earlier demand formula — catalog size × 0.08 plus monthly organic visits × 0.12, with a three-level exposure factor (1.0 / open, 0.7 / partial, 0.15 / major crawlers blocked). Reports built that way label the figures "legacy model", and they are not comparable with Radar-modeled ones.
Infrastructure Egress Drag
Estimated bandwidth consumed by automated scraping passes multiplied by standard cloud CDN edge egress rates ($0.08–$0.12 per GB).
Historical Corpus Floor
A modeled baseline acquisition price for complete historical archives ($0.02–$0.08 per article), based on disclosed terms in commercial licensing deals and collective bargaining filings.
Observed Inputs (measured, not modeled)
Two inputs are recorded by direct observation of the site rather than modeled. Both are reported on the economics tab of every report; neither is used in any figure above yet.
- Revenue model. The site's own markup is read for ad-serving tags (Google Publisher Tag, Prebid, Amazon UAM/TAM, Mediavine, Raptive/AdThrive, Ezoic, Freestar, generic ad-slot markup), subscription and paywall platforms (Piano, Memberful, Newspack metering, WooCommerce Memberships, Restrict Content Pro), and membership/donation platforms (Stripe, PayPal, GiveButter, Fundraise Up, Donorbox, Kindful, Classy) and donate or membership links in navigation. Two or more groups present is recorded as mixed, one group as that model, and none as undetermined. Source: the homepage and one sampled article page, as fetched for the audit — no additional requests.
- Pages in Common Crawl. The count of the site's pages present in Common Crawl's most recent crawl, read from its public index. Common Crawl is the largest public web corpus and the crawl most widely used to build AI training data; it is compiled by
CCBot, whose robots.txt status for the site is reported in the crawler access table. A site that disallowsCCBotis absent from, or only marginally present in, the corpus. The count is derived from the crawl index's own block structure (a full index block holds 3,000 records; interior blocks are full, edge blocks partial), so it is reported as an observed magnitude, not an exact page inventory. Source: Common Crawl's public index API (index.commoncrawl.org— collinfo and the CDX index), queried once per scan and cached per domain for 24 hours.
Macroeconomic Search Referral Displacement (Modeled)
Search referral displacement calculates the estimated annual digital revenue at risk as conversational answer engines provide direct answers without delivering downstream clicks.
The model synthesizes estimated organic search referrals, zero-click answer engine substitution rates, and blended digital RPM baselines into a single modeled annual revenue-at-risk figure.
- Incremental Zero-Click Shift: Reflects the modeled reduction in traditional organic search click-through rates specifically attributable to generative answers (modeled conservatively at 15%–35%). Note: This is distinct from baseline search behavior where queries end without clicks due to navigational or simple factual lookups.
- Blended Digital RPM: Valued at $12–$25 per 1000 pageviews, accounting for lost programmatic advertising, direct sponsorship impressions, and digital subscriber conversion opportunities.
Important Operational Note: Downstream search cannibalization is primarily driven by general search indexers (such as Googlebot). Configuring defensive rules against AI-specific training crawlers (such as Google-Extended) does not restrict Googlebot or prevent content from appearing in generative search features like Google AI Overviews.
Technical Readiness Scoring Rubric (0–100)
The composite Readiness Score (0–100) is a normalized, weighted index of three distinct sub-scores:
Composite Score = (Protection × 0.50) + (Visibility × 0.30) + (Legal/Policy × 0.20)Each dimension is independently evaluated and normalized on a 0 to 100 scale:
A. Protection Score (0–100, 50% Weight)
Evaluates whether robots.txt directives restrict access to commercial training crawlers and bulk commercial aggregators.
The engine evaluates access across major commercial foundation model trainers — including GPTBot, ClaudeBot, Google-Extended, CCBot, and ByteSpider — alongside bulk commercial aggregation services that reprocess published reporting at scale.
Informational (not scored): A separate class of link-preview and social unfurl fetchers is tracked for visibility but not counted toward the Protection score.
Scoring Mechanics: Points are deducted proportionately for each training or aggregation agent allowed uninhibited crawl access.
B. Visibility Score (0–100, 30% Weight)
Evaluates whether the property remains accessible to citation and search retrieval crawlers.
The engine evaluates accessibility for AI search and real-time citation agents — including PerplexityBot, OAI-SearchBot, Claude-SearchBot, and Applebot — together with their companion user-triggered retrieval agents.
Scoring Mechanics: Evaluates intentional crawl configuration. Sites receive deductions for unintended contradictions (e.g., blocking PerplexityBot while allowing Perplexity-User) or complete blacklisting that cuts off answer engine citation.
C. Legal & Policy Signals (0–100, 20% Weight)
Evaluates machine-readable copyright reservations and public AI documentation.
- Machine-Readable TDM Reservation: Evaluates presence of Text and Data Mining (TDM) reservations under EU DSM Directive Article 4 (via
robots.txt, W3C TDM Rep, or HTTP headers). - llms.txt Manifest: Detects presence of a canonical root
/llms.txtmanifest guiding AI agents to canonical reporting. - Public AI Governance Declaration: Assesses publicly accessible newsroom declarations governing external content use and AI disclosure standards.
Crawler list change (November 2026): DuckAssistBot, which DuckDuckGo uses to fetch pages for its AI-assisted answers, is now the ninth scored AI search and citation crawler. DuckDuckGo documents a robots.txt opt-out for it and says its data is not used to train AI models. The Visibility score is now out of nine crawlers; batches run before this change are marked as using an older scoring version. User-triggered agents that generally do not consult robots.txt (such as Google-Agent) are not scored, because a robots.txt rule would not reliably describe their access.
Note on the shared AI-training opt-out token (added November 2026): On October 30, 2026, OpenAI, Google, Microsoft and Anthropic said they will honor a single robots.txt group as a generative-AI training opt-out: User-agent: __usage_ai-training followed by Disallow: /. It is defined in an individual IETF Internet-Draft (not yet a ratified standard). As announced, Anthropic is implementing it now, OpenAI in the coming months, Microsoft from February 1, 2027, and Google during 2027.
How it is scored: When the group disallows /, the engine marks GPTBot, Google-Extended, ClaudeBot and anthropic-ai as Opted out (shared token) and credits them toward the Protection score, because their operators publicly committed to honor it. Microsoft has no separate training crawler in the tracked list. A crawler with its own explicit group keeps that group's result — the most specific rule wins. Rules under User-agent: * never count as a token opt-out. The token does not reach Meta, Apple, Amazon, Common Crawl, ByteDance or aggregator scrapers, so per-crawler rules are still needed for those.
Scope: The token covers training only. Google has said it will also treat it as an opt-out from Gemini grounding, which can reduce visibility in Gemini answers; Microsoft limits it to training and offers a separate grounding control. It does not affect Google AI Overviews or AI Mode, which use pages Googlebot crawls. Like every robots.txt rule, it is a stated preference, not technical enforcement.
Note on Content-Signal Directives: The engine parses Content-Signal from a line in robots.txt (Cloudflare's format — e.g. Content-Signal: ai-train=no), not from an HTTP response header. A declaration is recorded and shown in the report, but it contributes zero points to the Protection score. Protection is scored only from explicit per-crawler Disallow rules and the shared __usage_ai-training token above: Content-Signal is one site-wide statement rather than a per-crawler instruction, and no AI crawler operator has publicly committed to honor it.
Empirical Benchmarks
The assumptions in our economic models are calibrated against independent industry research:
| Dimension | Engine Assumption | Empirical Research Reference |
|---|---|---|
| Search Referral Contraction | 15%–35% substitution shift | Reuters Institute & Chartbeat: In the Reuters Institute’s Journalism, Media, and Technology Trends and Predictions 2026 (Newman, January 2026), Chartbeat data across 2,500+ news sites shows organic Google search traffic down 33% year-over-year (November 2024 → November 2025) globally, and 38% in the US. The same report’s executive survey separately projects a further 43% decline over the next three years. |
| Zero-Click Search Share | ~68% (modeled incrementally) | SparkToro / Similarweb, 2026 Zero-Click Search Study (U.S., Jan–Apr 2026): 68.0% of searches ended without a click. Prior baseline: SparkToro / Datos, 2024 Zero-Click Search Study: 58.5% U.S., 59.7% EU. Note: the two studies use different clickstream panels (Datos vs. Similarweb), so they are not a clean time series. |
| Newsroom Editorial Cost Inputs | Five tiers assigned by catalog size:
| Salary inputs: U.S. Bureau of Labor Statistics, Occupational Employment and Wage Statistics, May 2025, SOC 27-3023 (news analysts, reporters, journalists), median $62,200; NewsGuild-CWA published contract minimums (e.g., Austin American-Statesman 2024: $50,000–$60,000 by tenure; New York Daily News 2025: $60,000–$75,000 by role). The per-piece costs and tier totals used by the engine are derived assumptions built on these inputs, not published benchmarks. |
| Programmatic Scrape Pricing | $0.015–$0.1 per call | TollBit, Setting Rates documentation (2025): publishers set Summarization License rates per 1,000 pages benchmarked to their RPM, and Full Display License rates benchmarked to syndication rates. Engine floor of $0.015/call corresponds to a $15 RPM. The $0.100 ceiling is an engine assumption for full-display pricing, not an observed marketplace rate. No public rate card exists; the licensing marketplace remains underdeveloped. |
| Training Corpus Weight | Disproportionate representation | News/Media Alliance, White Paper and Technical Analysis on generative AI training (Oct. 30, 2023): news and digital media ranks third among source categories in Google's C4 dataset; curated training datasets overweight publisher content 5× to ~100× relative to its share of the web. Corroborating: Ziff Davis analysis of Common Crawl, C4, OpenWebText and OpenWebText2 (2024). |
Why the engine band sits below the observed decline. The 33% figure is the total observed year-over-year decline in organic search referrals, across all measured causes — search algorithm changes, Discover and social shifts, and seasonality — and is not decomposed into AI-attributable loss. The engine applies only the incremental share it attributes to generative answers, and deliberately discounts it: the displacement rate actually applied is 15%, the conservative end of the 15%–35% band shown above. The two figures are not directly comparable, and the engine’s is intentionally the more cautious.
Cohort Analysis vs. Individual Scaling
In macro portfolio dossiers (such as multi-title regional analyses), portfolio exposure is calculated by summing individual per-site modeled estimates, rather than applying synthetic linear multipliers.
Uncaptured licensing demand represents an unpriced transaction opportunity, whereas search cannibalization represents audience substitution. In our financial ledgers, these two metrics are tracked as separate line items and are never combined into a single interchangeable revenue figure.
Limitations & Scope of Audit
Users of this audit should consider the following methodological constraints:
- Passive Outside-In Analysis: The audit relies strictly on publicly observable edge signals (
robots.txt, HTTP response headers, root manifests, publicly rendered metadata). It cannot inspect origin server log files, private API gateways, or authenticated systems behind enterprise WAFs. - Modeled Estimates: All financial figures (licensing demand, egress costs, search displacement) are algorithmic estimates intended for comparative strategic analysis, not financial reporting or billing reconciliations.
- Private Commercial Agreements: Automated tools cannot detect private bilateral licensing or distribution agreements executed between publishers and AI technology platforms.
- Voluntary Adherence: Technical directives like
robots.txtrepresent voluntary standards. They do not constitute cryptographic encryption or absolute access prevention against bad-faith scraping actors. - Scanner Egress Origin: Automated scans originate from distributed edge nodes.
- Scanner Network Blocks: Some sites' firewalls block the scanner's network, so a scan may be unable to read
robots.txtor the homepage even though a visitor can. These are reported as "blocked from scanner network" and counted separately from general bot-protection blocks and site outages in the batch unassessed breakdown. - Scanner Request Inventory: For every submitted URL, the scanner requests the following paths on the target site:
/robots.txt(apex, www, and final host after redirect), the homepage (GET and HEAD),/tdmrep.json,/llms.txt, up to 3 article pages discovered from homepage links, AI policy page candidates (/ai-policy,/ai,/about/ai,/ethics/ai,/ai-guidelines), terms and privacy page candidates (/terms,/terms-of-service,/terms-of-use,/terms-and-conditions,/privacy,/privacy-policy), sitemap URLs fromrobots.txtSitemap directives, and the probe target (article or homepage) fetched once asGPTBotand once asClaudeBotfor AI-monetization gating detection. A full list is published on the bot info page. - Platform Detection from Page Assets: CMS and publishing architecture are passively inferred from standard public HTML asset references and document metadata.
How to read these figures. These are illustrative estimates from a revenue-per-visit model. They apply one blended value per thousand visits and assume lost search visits mean lost revenue. That fits ad- and sponsorship-supported sites best. For membership, donation or subscription organizations, the link between web traffic and revenue is different, and we are not aware of published measurements of AI's effect on those models. These figures are scenarios, not forecasts.
Independence & Disclosure
AI Readiness Check is an independent diagnostic tool designed to evaluate technical readiness and market exposure for digital news publishers. References to industry initiatives, collective licensing coalitions, or commercial clearinghouses (e.g., TollBit) are included strictly for analytical and educational benchmarking. AI Readiness Check maintains no commercial affiliation, equity interest, or exclusive partnership with the clearinghouse platforms referenced.
The training-corpus benchmark cited in Section 5 comes from the News/Media Alliance, with which the tool's creator is professionally affiliated; it is cited as an industry source, and corroborating independent analysis is listed alongside it.
Ready to benchmark your site? Run a check or learn more about the local-news viability ecosystem.