Changelog

Scoring

Crawler checks rebuilt around citation, not training

Every major AI vendor runs more than one crawler, and they do different jobs. Until now we checked only the best-known one from each — and in two cases out of three, that was the crawler with nothing to do with whether you get cited.

OpenAI trains on your content with GPTBot. It cites you through OAI-SearchBot, which builds the index behind ChatGPT answers, and ChatGPT-User, which fetches your page live mid-conversation. Blocking GPTBot — as many sites deliberately did — never affected ChatGPT citations, yet we reported those sites as invisible to ChatGPT. They were not.

The reverse mattered more: a site that allowed GPTBot but blocked OAI-SearchBot genuinely could not appear in ChatGPT, and we told it everything was fine.

Your score now reflects the search and live-fetch agents for each vendor. Blocking a training crawler is a legitimate business decision and no longer costs you points — it is reported so you can see it, not graded.

Corrected guidance

Google-Extended does not control AI Overviews

We told you that unblocking Google-Extended would get you into Google AI Overviews. That was wrong, and it had spread to five places across the report and our guidance pages. All five are corrected.

Google-Extended governs training and grounding for Gemini and Vertex AI. AI Overviews is part of Google Search and is crawled by Googlebot. Blocking Google-Extended will not remove you from AI Overviews, and unblocking it will not add you.

The check remains in your report as an informational note rather than a scored item, because opting out of Gemini training is a decision, not a defect. If you acted on our earlier advice and unblocked Google-Extended expecting AI Overview placement, that change had no effect on it either way.

Accuracy

We now read robots.txt the way crawlers do

The old reader asked one question: is the site root blocked? A robots.txt that left the root open while disallowing /blog/ — where all the content lives — passed cleanly.

It now tests the pages we actually crawled against your rules and names the blocked paths, matches wildcard and end-anchor patterns, applies the correct precedence so a narrow Allow can carve an exception out of a broad Disallow, merges repeated sections for the same agent instead of ignoring all but the first, and treats a bare Disallow as permitting everything.

A robots.txt returning a server error is now distinguished from a missing one. Under RFC 9309 the first tells crawlers to treat your whole site as off limits, which is an outage rather than an absent file.

Scoring

A missing robots.txt is one finding, not five

If you had no robots.txt, we failed you for the missing file and then failed each crawler check on top of it — the same problem charged five times, worth roughly nine points.

With no robots.txt, every crawler is allowed. That is how the standard works. You now get a single finding for the missing file, and the crawler checks pass.

The same applies to agents you simply do not mention. Not naming a crawler permits it, so we no longer deduct points for a configuration that follows the specification.

Scoring

Checks that cannot apply no longer move your score

Several checks were quietly awarding or deducting points when they had nothing to measure. A single-language site collected a free point for hreflang tags it had no reason to add. A site with no images collected full marks for image optimisation. In the other direction, a site lost two points on canonical tags when our crawl came back empty — a problem on our end, charged to you.

Fifteen checks now sit out of the calculation entirely when they do not apply, instead of scoring as a pass or a fail. That includes cases where our crawler, rather than your site, is the limitation: if we cannot reach enough of your pages to judge internal linking, content length, title tags or crawl depth, we say so instead of scoring you zero.

One deliberate exception: an llms.txt containing no links still counts against you. That file exists to point models at your content, so an empty one is a real shortfall rather than an unmeasurable case.

Scoring

Two signals were scored far above their real worth

entities.txt was worth six of the nineteen points in AI Agent Readiness — nearly a third of the category — and the fix we recommended was to generate the file with our own tool. It is our own format. No crawler reads it. It is now reported for reference and scored at zero, and we no longer recommend creating one as a fix.

FAQ and HowTo schema were six of twenty-seven structured-data points, flagged high priority as though their absence cost you a search feature. Google retired both rich results in 2023 and 2024. They are now worth two points each, marked as suggestions, and the report says plainly that the remaining value is AI answer extraction rather than search appearance.

Accuracy

Seven checks were measuring the wrong thing

URL friendliness counted every query parameter, so a normal tracked marketing link carrying four UTM tags tripped the "too many parameters" rule on its own. Tracking parameters are now excluded.

Internal linking counted every link on the page, so a repeated navigation menu cleared the bar by itself. It now counts links inside your content only.

Crawl depth counted slashes in your URLs, so a post at /blog/2026/09/my-post was flagged as four levels deep even when it sat one click from your blog index. It now walks your internal links from the homepage and counts actual clicks — and reports pages with no internal path from the homepage at all, which is more serious than being far from it and was previously invisible.

Alt text treated alt="" as missing. That is the correct way to mark a decorative image so screen readers skip it, and counting it as a fault pushed sites toward writing filler text on spacers and icons.

The content ratio counted inline SVG icons and inline CSS as page weight. On one site tested those were ninety percent of the file, producing a 0.5% score for a page full of content; excluding them moved it to 14.6%.

Short titles are no longer marked as failures. "Pricing – Vercel" is sixteen characters and perfectly good, but the old rule flagged twenty-two of twenty-four pages on that site. A missing title is still a failure; a short one is now a suggestion.

Core Web Vitals now use the real-user measurements Google collects from Chrome, falling back to a lab test only when your site has too little traffic for Google to report on. This is the same data Search Console grades you against, so your report and Search Console should now agree. Responsiveness also stops judging one metric against another metric’s thresholds.

New checks

Fourteen new checks

Near-duplicate content compares page text with the navigation and footer stripped out, so it catches near-identical pages — the same template with one city or product name swapped — that exact matching never saw.

Breadcrumb schema, still rendered by Google in search results. Article schema, checking the headline, author, publication date and publisher that AI engines read for authority and freshness. WebSite schema with a search action. Consistency between your organisation entities, which catches a CMS plugin injecting a second, conflicting block.

Canonical header conflicts, where a canonical arrives in an HTTP header as well as a page tag and the two disagree. Canonical scope, covering cross-domain canonicals and paginated pages that collapse back to page one. Sitemap size limits, past which search engines reject a sitemap whole. Sitemap lastmod validity, since a date that will not parse is a recrawl signal silently thrown away.

Transport and caching headers, reported as one finding rather than three so a single misconfigured server is a single item on your list. Render-blocking resources, naming the specific files holding up your first paint. Crawl-delay, which can quietly cap how much of a large site ever gets fetched. Agent discovery endpoints, looking for an MCP descriptor — how an AI agent finds your tools without a person reading your docs.

And microdata and RDFa, reported rather than scored: every other structured-data check reads JSON-LD only, so a site using microdata scored as having no structured data at all.