Platform pages checked on 3 November 2026. We update this guide when they change.
Every big AI platform has now written down how its systems read websites. Google, OpenAI, Anthropic, Perplexity, Microsoft, Apple and Meta all publish pages about their crawlers, and lists of these free guides are going around LinkedIn as the alternative to paying for an AEO course.
Read them. They’re free, mostly clear, and some of what they say surprises people who run websites for a living. But they all answer the same question: can the assistant get in and read the page? The question that decides the sale comes after that.
The platform guides tell you how to get in. None of them tells you what the assistant says about you once you're in.
This guide covers both. First, what the platforms say, in one place. Then what we find when we audit real brands’ sites, where the problems are rarely the ones the guides warn about. And last, the part no guide can cover.
What Google says about AI Overviews and AI Mode
Google’s page on its AI features is short and direct: “There are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary.” A page has to be indexed and eligible to show a snippet in Search, the same as before.
It also says you don’t need new machine readable files, AI text files or special structured data. So an llms.txt file or extra schema won’t get a page into Google’s AI answers.
Three details on the same page matter more than the headline:
- One question becomes several searches. Both features may use what Google calls query fan-out, running related searches across subtopics to build one answer. The buyer asks once, and the answer can draw on pages found for questions the buyer didn’t type.
- The controls are the old ones. nosnippet, data-nosnippet, max-snippet and noindex limit what Google shows from a page. Google-Extended, the token many sites block, has no effect on Search: it covers Gemini’s training and its grounding in the Gemini apps, Google says.
- You can’t see it in Search Console. Clicks from AI Overviews and AI Mode are counted inside the ordinary Web search type, with no filter of their own. Google calls these clicks “higher quality” but publishes no figure, while Seer Interactive reports organic click through falling by about 61% on searches that show an AI Overview.
Every assistant sends more than one bot
The AI companies split their crawlers by job, and each job behaves differently:
- A search crawler builds the index the assistant searches when it answers. Block it, and you lose your place in that assistant’s search answers.
- A user fetcher opens a page because someone asked the assistant to, right then.
- A training crawler collects text that may go into a future model. Block it, and new content on your site won’t reach the next model’s memory, while you stay in the search answers.
Each company runs separate bots for search, for user requests and for training.
The names to use in robots.txt, as each company describes them. Read at source on 3 November 2026.
| Company | Builds the search index | Opens a page when a user asks | Collects for training |
|---|---|---|---|
| OpenAI | OAI-SearchBot | ChatGPT-Userrobots.txt “may not apply” | GPTBot |
| Anthropic | Claude-SearchBot | Claude-Userfollows robots.txt, like all three | ClaudeBot |
| Perplexity | PerplexityBot | Perplexity-User“generally ignores” robots.txt | None namedPerplexityBot isn’t used for training |
GooglebotSearch, AI Overviews and AI Mode | Not covered here | Google-Extendeda token, not a crawler; also covers grounding in the Gemini apps | |
| Apple | ApplebotSpotlight, Siri and Safari | Not covered here | Applebot-Extendeda token, not a crawler |
| Meta | Meta-WebIndexerMeta AI search | Meta-ExternalFetcher“may bypass” robots.txt | Meta-ExternalAgentalso indexes content |
Swipe the table sideways to see every column.
The crawler pages of OpenAI, Anthropic, Perplexity, Google, Apple and Meta, linked under the article.
Three things in that table catch people out.
The training and search bots are separate. Blocking GPTBot or ClaudeBot keeps new content out of future training and leaves you in ChatGPT’s and Claude’s search answers. OpenAI says its settings are independent of each other, and that a change to robots.txt takes about 24 hours to reach its search.
Blocking a training bot leaves you in the answers. Blocking a search bot takes you out of them.
robots.txt doesn’t stop every visit. OpenAI says its rules “may not apply” to ChatGPT-User, because a person asked for the page. Perplexity says its user fetcher “generally ignores” robots.txt, and Meta says its fetcher “may bypass” it. Anthropic says all of its bots follow robots.txt. So robots.txt chooses which jobs you take part in. It can’t keep a page away from every assistant.
Two of the names aren’t crawlers. Google-Extended and Applebot-Extended are tokens. They tell Google and Apple how they may use what Googlebot and Applebot already collected, and blocking them doesn’t remove a site from Google’s or Apple’s search.
Microsoft: the assistant reads your page in pieces
Microsoft’s Bing team wrote the most practical of the guides. Bing’s index powers Copilot, and Microsoft says assistants like Copilot don’t read a page from top to bottom. They break it into pieces, judge each piece for authority and relevance, and build one answer from pieces of several sources.
It also names what gets missed: details inside tabs or expandable menus that may not be rendered, facts kept in PDFs, and details that only exist in images. And it says that there’s no guaranteed way to be selected.
What we find when we audit sites
The guides describe how things should work. When we audit a brand’s site, we test what each crawler is actually served, page by page. These are patterns from the brands we’ve studied, anonymised.
The robots.txt you see isn’t always the one that’s served
On 15 September 2026, Cloudflare replaced its single “Block AI Bots” switch with three settings, Search, Training and Agent, and moved the sites that had the old switch on over to the new ones without asking. Cloudflare can also serve its own managed robots.txt in front of the site’s file. So the robots.txt in your CMS can differ from the one the crawlers read.
One line can close the pages that carry your facts
One supplement brand’s robots.txt closed a whole folder to every crawler. The folder held the pages about the founder, the advisors and the vets, and the only page that said when the company was founded. No crawler could read that fact, whether it came for search, for training or for a buyer’s request.
Key content only arrives after JavaScript
On a donation brand’s site, the page listing its drop-off centres sent 28 words in its HTML and 229 once JavaScript had run: 88% of the page arrived only after the script. A crawler that doesn’t run JavaScript gets the page without the list. None of the crawler pages above says whether its bot runs JavaScript, so the safe place for a fact is the HTML the server sends.
The markup tells machines something the page doesn’t
On a donation service’s site, the structured data on every page we sampled named its partner charity as the organisation, and marked the service pages up as blog articles. A person reading the page saw one company. A machine reading the markup saw another.
The assistants repeat what’s broken
Searching assistants read the small print. ChatGPT passed a placeholder protection number, “XXX”, from a tour operator’s booking terms on to a buyer, along with a balance due date that differed between the site and the terms.
Absence is penalised
Across the brands we’ve studied, a brand with too little about it online didn’t get a neutral answer. A tour brand was ranked last “for lack of evidence”. An agency was ranked last for “not enough evidence”.
When the search finds nothing about a brand, the assistant doesn't say it doesn't know. It ranks the brand last.
Our own tests can be wrong too
The first time we ran our audit on one site, it reported two AI crawlers refused on every page. The host’s panel blocked nothing: the refusals were a throttle on our own imitation traffic. A test that pretends to be a crawler can show a block that isn’t there, so we confirm a block in the host’s settings or logs before we call it one.
What files and markup don’t do
A lot of what’s sold as AI optimisation is files and markup: llms.txt, more schema, FAQ blocks. Google says none of it is needed for its AI features. Microsoft recommends structured data, clear headings and lists, so the pieces are easy to label and lift.
The published tests point to the content. The best known study of generative engine optimisation, by Aggarwal and colleagues in 2024, reports gains from adding citations, quotations and statistics to the text, and a loss from keyword stuffing. It didn’t test schema.
Our view: markup helps a machine find and label a page. What the answer says about you comes from the content, on your pages and on everyone else’s.
What none of the guides can tell you
Everything above decides whether an assistant can read you. None of it decides what the assistant says once it has.
In our studies, being read and being picked are often far apart. One bookstore’s own site was among the sources the assistants cited most, while they rarely chose it when the buyer decided. An insurer lost most of its picks to the findings of one consumer survey. Threads on Reddit and Mumsnet told buyers not to subscribe to a recipe app.
Being read is the price of entry. Who gets picked depends on what the assistant has read about you, much of it on pages you don't own.
None of the platform pages can show you that, and Search Console can’t either. You find it by asking the assistants the questions your buyers ask, from the first question to the decision, and reading what they say about you and your competitors.
Ads work the same way. Google says advertisers can’t opt out of ads in AI Overviews, and OpenAI’s ad bot reads an advertiser’s landing page to decide when the ad is relevant. The answer above the ad is built from everyone else. We wrote about that in ChatGPT ads lose even to ChatGPT’s own answers.
A short checklist
Before you change a word of content, check that:
- The robots.txt served on your live domain lets in the search bots of the assistants your buyers use, and matches the copy in your CMS.
- Your CDN and host settings agree with it. Cloudflare’s Search, Training and Agent settings change what the crawlers get.
- Every page that states a fact about you is open and indexable: about, founder, pricing, reviews, policies and contact.
- Those facts are in the HTML the server sends, outside tabs, PDFs and images, and they don’t wait for JavaScript.
- Your structured data says the same as the page: the right company, prices and address.
- No placeholder text or conflicting figures sit anywhere a search can reach, terms and conditions included.
- You’ve made the training decision on its own. Blocking training keeps new content out of future models and leaves your search presence as it is.
- You check again after every launch, migration or CDN change, and you date it. Google’s crawler page was updated on 30 October and Apple’s in September.
That’s the floor. It gets you read.
Every study we run includes a technical audit of the brand’s site, because a closed door explains some results and decides which fixes can work. Then we research what the AI assistants tell buyers about the brand and its competitors, from the first question to the decision. If you want that for yours, you can request a free assessment.
Sources: Google, "AI features and your website", last updated 10 December 2025, and "Google's common crawlers", last updated 30 October 2026. OpenAI, "Overview of OpenAI crawlers". Anthropic, "Does Anthropic crawl data from the web, and how can site owners block the crawler?", updated 7 April 2026. Perplexity, "Perplexity crawlers". Apple, "About Applebot", 4 September 2026. Meta, "Meta Web Crawlers". Krishna Madhavan, Microsoft Bing, "Optimizing Your Content for Inclusion in AI Search Answers", 8 October 2025. Google Ads Help, "About ads and AI Overviews". Cloudflare, blog post of 15 September 2026. Seer Interactive, AI Overviews and click through, 2026 update. Aggarwal and others, "GEO: Generative Engine Optimization", KDD 2024. Every platform page was read on 3 November 2026. The audit examples come from our own work, anonymised.
Changes: 3 November 2026, first version.