Crawler and Index Eligibility for AI Search

Eligibility problems sit below the content layer. A page may be accurate, useful and well linked while a crawler receives a block, an error response, a duplicate URL, an empty app shell or an indexing restriction. A sound review follows the request all the way to the document that an engine can process.

Google makes the threshold explicit for its own AI search features: a supporting page must already be indexed and eligible to appear in Google Search with a snippet. Google lists no extra technical requirement for AI Overviews or AI Mode (Google Search Central). That puts familiar crawl and index controls at the start of a GEO investigation.

Eligibility has five gates

Treat the path into search and answer surfaces as a sequence:

1. Fetch permission: the named crawler can request the URL under the applicable robots.txt policy.
2. Response handling: the server returns a meaningful status and a viable final URL.
3. Canonical selection: page, header, redirects, internal links and sitemap point towards the intended URL.
4. Content availability: the important copy and links survive the rendering path used by the crawler.
5. Index and preview permission: robots meta directives and response headers permit indexing and the required preview.

A green result at one gate says little about the next. A robots rule can allow a fetch that ends in 403. A 200 response can carry noindex. An indexable page can declare a different canonical. A correct canonical can lead to an app shell with no meaningful body in the initial response.

Classify the crawler before changing policy

The label “AI crawler” hides separate jobs. Current vendor documentation distinguishes model-development collection, search indexing and retrieval initiated by a person.

OpenAI’s crawler documentation says GPTBot may collect content for training, OAI-SearchBot supports ChatGPT search results, and ChatGPT-User is used for certain user actions rather than automatic web crawling. OpenAI also says the GPTBot and OAI-SearchBot settings are independent.

Anthropic’s crawler documentation makes the same three-way distinction for ClaudeBot, Claude-SearchBot and Claude-User. It says its bots honour standard robots.txt directives. OpenAI notes that robots.txt rules may not apply to ChatGPT-User requests because a person initiated them. Preserve that vendor-specific difference in policy and reporting.

When the question is limited to a named bot and release, run an AI crawler access check. Continue through the full trace here when the problem may sit in redirects, canonicals, rendering or index controls.

Trace the request from the edge

Follow one URL across all five gates

The following trace is illustrative. The URL, observations and artefacts are fictional and show the fields an operator could record; they are not a live result.

The value lies in the evidence chain. “Allowed” is an interpretation. The retrieved file, matched user-agent group, response headers and document extracts make it reviewable.

Read robots rules as fetch controls

Google describes robots.txt as a way to manage crawler traffic and URL access. It also warns that a disallowed URL can still appear in results when other pages link to it, because the crawler may know the URL without fetching its content (Google robots.txt guide).

That distinction changes the audit:

• Note the exact user agent under test.
• Retrieve robots.txt from the relevant host and protocol.
• Apply the longest matching rule under that crawler’s documented interpretation.
• Check each subdomain separately.
• Preserve deliberate differences between training and search agents.

Security controls belong elsewhere. Authentication, authorisation and network access protect private material. Cooperative crawler directives express preferences to agents that honour them.

Follow status and redirects to the final URL

The response code is the next gate. Google says a 2xx response can enter processing, while indexing remains a separate decision. Permanent redirects act as strong signals towards the target, and 4xx or 5xx responses represent errors in Search Console (Google’s HTTP status guidance).

Capture the entire redirect chain. A single request to the starting URL can miss:

• an HTTP-to-HTTPS redirect that lands on the wrong host;
• a locale or device redirect that changes the content;
• a loop between trailing-slash variants;
• a CDN challenge returning 403 to a verified crawler;
• a soft error page served with 200.

The final response body matters as much as the final code. A successful status attached to an empty template gives the indexing pipeline little to process.

Reconcile canonical signals

Canonicalisation tells search systems which URL represents a duplicate group. Google recommends an absolute, self-referential canonical on the preferred page and warns against conflicting canonical methods (Google canonical guidance).

Compare:

• the final URL after redirects;
• the HTML canonical;
• any Link canonical response header;
• sitemap entries;
• internal links;
• language or regional alternates.

An audit should surface disagreement instead of guessing which signal will win. A canonical is a hint within a wider cluster of signals. Search systems can select another URL when the evidence conflicts.

Compare the initial and rendered document

Google can render JavaScript with headless Chromium after fetching a successful page. Its documentation explains that server-rendered pages expose content in the first HTML response, while app-shell pages require a later rendering step (Google JavaScript SEO basics).

Save both states where JavaScript is involved. Compare the page title, canonical, main heading, core answer blocks, internal links and robots meta tag. Flag content that appears only after user interaction, fails under a clean browser profile or changes identity during hydration.

Google’s rendering capability does not describe every search or AI agent. Stable HTML for the main content reduces dependence on crawler-specific JavaScript execution.

Inspect index and preview directives together

Robots meta tags and X-Robots-Tag headers can control indexing and previews. Google explains that these directives must be fetched before they can be read; a URL blocked in robots.txt may hide its noindex instruction from Google (Google robots meta documentation).

Review page-level and header-level controls for:

• noindex;
• nosnippet;
• max-snippet;
• data-nosnippet;
• agent-specific directives;
• conflicting values set by the CMS, CDN or application.

Preview limits express a publishing choice. They can also narrow eligibility for Google’s AI features, since supporting links must be eligible for a snippet.

Record the tested URL, controls and owner

The handover should contain the tested agents, the policy intent, raw responses, final URLs, canonical comparison, initial and rendered extracts, index controls, unresolved conflicts and an owner for each change. Include the verification method beside every status. The 14-day GEO sprint shows how to package that trace with deployment checks and an owner handover.

That record gives the next release a baseline. When visibility changes, the operator can separate a policy change from a routing fault, a render regression or an indexing decision without reconstructing the whole path from memory.