The AI-crawler checklist I run before a new site goes live
AI crawler policies, bot lists, and platform defaults change quickly. This is a snapshot from October 2026, based on what I run on my own sites today. Re-check the official lists before you copy any of it.
Since July 2025, Cloudflare blocks AI crawlers on newly onboarded domains by default. If you do nothing, a new site on Cloudflare is simply absent from ChatGPT, Claude, and Perplexity — and nothing errors, nothing warns you, and your analytics stay silent because the traffic never arrives. Here is the four-step checklist I now run on every new static site before I consider it launched.
New Cloudflare domains hide from AI crawlers by default
This is the trap, and it is silent. Cloudflare turned on default AI-crawler blocking for new domains on 2025-07-01 (Cloudflare, 2025). An existing site keeps its settings; a new domain inherits the block. Your pages render fine, Google Search Console shows nothing unusual, and the first symptom is that AI assistants simply never mention your site — a symptom with no dashboard.
The crawl access layer is worth treating as launch infrastructure, same as DNS and SSL. I missed this on exactly zero sites only because the family SOP caught it before launch — the checklist below is that SOP.
Step 1: Explicitly allow search-class crawlers in Cloudflare AI Crawl Control
Cloudflare’s AI Crawl Control is the switch that matters most. Go in and explicitly allow the search-class crawlers — the bots that fetch pages to answer user queries: OAI-SearchBot (ChatGPT search), ClaudeBot (Claude), and PerplexityBot (Perplexity). These are the ones that make your content quotable inside AI answers.
Also allow the user-triggered fetchers — bots that fetch a page because a user explicitly asked an AI to read it. Google’s list (updated 2025-10/11) includes Google-NotebookLM, Google-CWS (Chrome’s AI features), and Google-Pinpoint. Being read on demand is the same value as being read by search bots; block it and “summarize this page for me” fails for your domain.
| Crawler class | Examples | Why it fetches | My setting |
|---|---|---|---|
| Search-class AI bots | OAI-SearchBot, ClaudeBot, PerplexityBot | Answer user queries with your content | Allow |
| User-triggered fetchers | Google-NotebookLM, Google-CWS, Google-Pinpoint | A user asked an AI to read this page | Allow |
| Training-class crawlers | GPTBot, Google-Extended | Model training corpora | Allow (optional — I allow them) |
Step 2: Name the bots in robots.txt — and skip invented fields
robots.txt still works, but only for documented fields. Google has said since 2024-10 that it ignores undocumented self-invented directives like NoAI (Google, 2024). Writing NoAI: false or similar into robots.txt controls nothing anywhere. Control goes through the official mechanisms: Cloudflare AI Crawl Control for the edge, and named user-agent groups in robots.txt for everything else.
On my own sites the robots file names the AI crawlers explicitly — ten user-agent groups, every one of them allowed — and a contract test fails the build if any group gains a Disallow. Named groups beat wildcards here: you can see exactly who you admitted, and a diff on the file is a readable audit trail.
Step 3: Keep llms.txt, but do not count it as a Google lever
Google’s official guidance across three documents (2025-05 generative AI content guidance, 2025-12 AI features documentation, and the 2026-05 AI optimization guide) settles two questions: there is no separate AI index to optimize for, and llms.txt is not counted by Google Search. GEO, in Google’s own framing, is traditional SEO done well — crawlable pages with citable, quotable content.
I still ship /llms.txt on every site, with this blog’s own copy covering the hub and main pages. It costs almost nothing and non-Google consumers use it. But I keep it out of the ROI story: for Google, its expected value is zero, and pretending otherwise would distort what I measure.
Step 4: Reconcile with the GSC Generative AI report every month
Since 2026-06, Search Console has a Generative AI report showing how often your content appears in AI experiences (Google, 2026). Two things to know before you read it: it reports impressions, not clicks, and it has no query dimension. You can see that AI surfaces showed your page, not which question triggered it.
This report is now part of my monthly inspection routine, next to classic queries and pages. One related setting worth leaving alone: the “Search generative AI” participation control (opened globally 2026-08-31) is not a ranking lever in either direction — opting out does not protect rankings and opting in does not boost them. I keep participation on, because for a content site AI visibility is the point.
What I verified on my own sites, and what I did not
On this blog the full stack went live on 2026-09-29: robots.txt with ten named AI-crawler groups all allowed, /llms.txt covering the hub and main pages, a Cloudflare zone audit showing no AI blocking, and curl checks confirming every rule served what I intended. Contract tests lock the robots file so a regression fails the build instead of slipping out. The same SOP now runs across my family of static tool sites — the newest of them went through the same checklist on launch day.
The honest boundary: all of this verifies crawl access, not outcomes. I can prove the bots are admitted; I cannot yet point at traffic and say the checklist caused it. That part needs months of the GSC Generative AI report before anyone — including me — should claim it works.
The one-line version
Allow search-class and user-triggered AI crawlers in Cloudflare AI Crawl Control, name them in robots.txt with no invented fields, keep llms.txt as a free extra with zero Google ROI, and reconcile against the GSC Generative AI report monthly.
This article was created with the help of AI. AI was not used to write the content; it assisted only with translation and grammar checks.