Check which AI crawlers can reach your site
Checks your robots.txt against 22 AI user-agents, then fetches your page as OAI-SearchBot, GPTBot, ClaudeBot and PerplexityBot to catch CDN/WAF blocks, measures how much text needs JavaScript, and gives you a prioritized fix list. Free, nothing stored.
Recommended robots.txt block for AI visibility
That is the free check. Want the full picture?
The tool tells you whether AI crawlers can reach you. It cannot tell you why ChatGPT recommends a competitor. That is what our GEO team does:
- Prompt audit: which questions in your category ChatGPT, Perplexity, Gemini and Google AI Overviews answer, and which brands and sources they cite
- Technical fixes: crawler access, rendering, llms.txt, structured data and page structure that answer engines can quote
- Citable content: comparison, alternative and FAQ pages built around the prompts buyers actually ask
- Third-party evidence: genuine, disclosed participation on Reddit, forums and review sites that AI answers lean on
Short answer
An AI crawler checker tells you whether ChatGPT, Claude, Perplexity, Gemini and Copilot can actually read your site. Two things decide it: your robots.txt rules for each AI user-agent, and whether your server or CDN serves those bots the same HTML as a browser. The free checker on this page tests both, for 22 AI user-agents, with no sign-up.
Search bots vs. training bots: the split that matters
Every major AI company now runs separate crawlers for search (fetching pages so answers can cite and link them) and training (collecting data for future models). You can block training and still be cited in AI search. The reverse mistake is common: sites block everything that sounds like “AI” and quietly drop out of ChatGPT and Perplexity answers.
| User-agent | Operator | What it does | If you block it |
|---|---|---|---|
OAI-SearchBot | OpenAI | Indexes pages for ChatGPT search results and citations | You can disappear from ChatGPT search answers |
ChatGPT-User | OpenAI | Fetches a page when a user asks ChatGPT about it | ChatGPT cannot open your page on request |
GPTBot | OpenAI | Collects data for model training | Nothing in ChatGPT search (separate bot) |
Claude-SearchBot / Claude-User | Anthropic | Search indexing / user-requested fetches | Fewer Claude citations of your pages |
ClaudeBot | Anthropic | Model training | Nothing in Claude search |
PerplexityBot / Perplexity-User | Perplexity | Index for answers / user fetches | Perplexity cannot cite you |
Googlebot | Search, AI Overviews and AI Mode | You leave Google entirely | |
Google-Extended | Controls use in Gemini training and grounding | Not used for Search ranking or AI Overviews | |
Bingbot | Microsoft | Bing and Copilot answers | You lose Bing and Copilot citations |
CCBot | Common Crawl | Open web dataset used by many model builders | Less presence in future training sets |
Tokens from each operator’s published crawler documentation. The checker above tests 22 user-agents.
How the checker reads your robots.txt
The checker follows the robots.txt standard (RFC 9309) and Google’s documented behaviour, so its verdict matches what well-behaved crawlers do:
- A named group replaces the * group. If you have
User-agent: GPTBot, GPTBot ignores everything underUser-agent: *. - The longest matching rule wins.
Allow: /blog/beatsDisallow: /for blog URLs. - On a tie, Allow wins.
- Wildcards
*and end anchors$are supported. - No robots.txt (404) means allow everything. A 5xx error makes Google and others treat the whole site as blocked until it is fixed.
Use the path option to test a specific URL, for example a product page behind a Disallow: /products/*? rule, or paste a draft robots.txt to test it before it goes live.
Common mistakes that silently block AI search
- A staging rule that shipped:
User-agent: *+Disallow: /left over from development blocks every AI crawler, not just Google. - Blocking GPTBot to “stay out of ChatGPT”. GPTBot is the training crawler. ChatGPT search uses OAI-SearchBot and ChatGPT-User. Decide on each separately.
- Adding a named group and forgetting your private rules. Once a bot has its own group, your
Disallow: /admin/lines under*no longer apply to it. Copy them in. - CDN-level AI bot blocking. Cloudflare and other CDNs offer one-click AI crawler blocking, and some turn it on by default for new sites. It overrides a permissive robots.txt.
- Content that only exists after JavaScript runs. Most AI crawlers do not execute JavaScript, so client-rendered prices, specs or reviews can be invisible even when access is allowed.
- Expecting Google-Extended to affect AI Overviews. It does not; AI Overviews and AI Mode use normal Googlebot access.
The live access test: what robots.txt cannot tell you
A permissive robots.txt only says crawlers are allowed. The live test checks whether your server actually serves them. It requests the page five times through a neutral fetch service: once as a normal browser (the baseline) and once each with the official user-agents of OAI-SearchBot, GPTBot, ClaudeBot and PerplexityBot. Then it fetches a fully rendered copy and compares the text.
| Result | What it means | Usual fix |
|---|---|---|
| Blocked (401/403/429/503) | Your CDN, firewall or host refuses that user-agent. | Cloudflare: Security → Bots, AI Crawl Control, WAF custom rules, rate limiting. Other hosts: bot management or firewall rules. |
| Challenge page | The bot gets a “verify you are human” page instead of your content. | Skip the challenge for verified AI search bots, or lower the security level on content pages. |
| Thin content | The bot gets far less text than a browser. | Serve bots the same HTML as users; check consent walls and bot-specific templates. |
| Low text in raw HTML | Most of the text appears only after JavaScript runs. | Server-render (SSR/SSG) or prerender product facts, prices, specs and FAQs. |
One limit to know: the requests come from the fetch service’s IP addresses, not OpenAI’s or Anthropic’s. A “Blocked” result is a strong signal, because it means the user-agent alone is refused. An “OK” result shows the user-agent is not refused, but a CDN that verifies crawler IPs could still treat the real bot differently, so confirm in your server logs or CDN analytics.
The report also checks whether robots.txt declares a sitemap and links to a Bing site: search. Copilot answers from Bing’s index, and ChatGPT search has been reported to use it too, so pages missing from Bing are harder to cite. Submit your sitemap in Bing Webmaster Tools and enable IndexNow to speed that up.
What to allow: a sensible default
For most brands that want to be recommended in AI answers, the default should be: allow every AI search crawler, keep your private paths disallowed, and make a deliberate business decision about training crawlers. Publishers with licensing value often block training; product and service brands usually benefit from being in training data too, because models learn which brands belong to a category.
The snippet generator in the tool writes both versions. Paste it above your existing rules, copy your private Disallow lines into the AI group, then re-run the check.
When access is fixed, the next step is being worth citing: how to rank in ChatGPT search, how to get cited by Perplexity and a free llms.txt generator.
All 22 AI crawler user-agents the checker tests
The full list below is what the tool checks against your robots.txt. Use the exact token in a User-agent: line; tokens are not case-sensitive.
| robots.txt token | Operator | Type | Used for | Default advice |
|---|---|---|---|---|
OAI-SearchBot | OpenAI | AI search / user fetch | ChatGPT search | Allow if you want to be cited |
ChatGPT-User | OpenAI | AI search / user fetch | ChatGPT user fetch | Allow if you want to be cited |
Claude-SearchBot | Anthropic | AI search / user fetch | Claude search | Allow if you want to be cited |
Claude-User | Anthropic | AI search / user fetch | Claude user fetch | Allow if you want to be cited |
PerplexityBot | Perplexity | AI search / user fetch | Perplexity index | Allow if you want to be cited |
Perplexity-User | Perplexity | AI search / user fetch | Perplexity user fetch | Allow if you want to be cited |
Googlebot | AI search / user fetch | Search + AI Overviews / AI Mode | Allow if you want to be cited | |
Bingbot | Microsoft | AI search / user fetch | Bing + Copilot | Allow if you want to be cited |
Applebot | Apple | AI search / user fetch | Siri / Spotlight / Safari | Allow if you want to be cited |
DuckAssistBot | DuckDuckGo | AI search / user fetch | Duck.ai answers | Allow if you want to be cited |
MistralAI-User | Mistral | AI search / user fetch | Le Chat user fetch | Allow if you want to be cited |
Meta-ExternalFetcher | Meta | AI search / user fetch | Meta AI user fetch | Allow if you want to be cited |
GPTBot | OpenAI | Training | Model training | Business decision |
ClaudeBot | Anthropic | Training | Model training | Business decision |
anthropic-ai | Anthropic | Training | Legacy token | Business decision |
Google-Extended | Training | Gemini training & grounding | Business decision | |
Applebot-Extended | Apple | Training | Apple model training | Business decision |
CCBot | Common Crawl | Training | Open dataset used by many LLMs | Business decision |
Meta-ExternalAgent | Meta | Training | Model training | Business decision |
Bytespider | ByteDance | Training | Model training | Business decision |
Amazonbot | Amazon | Training | Alexa / Amazon models | Business decision |
cohere-ai | Cohere | Training | Model training | Business decision |
Tokens and purposes from each operator’s published crawler documentation (OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, DuckDuckGo, Mistral, Meta, Common Crawl, ByteDance, Amazon, Cohere). anthropic-ai is a legacy token kept for older robots.txt files.
How this checker compares with other AI crawler checkers
Most free AI crawler checkers only parse robots.txt. That misses the two most common real-world blocks: CDN/WAF bot rules and JavaScript-only content. Here is what to look for when you pick one.
| Check | This checker | Typical robots.txt-only checker |
|---|---|---|
| robots.txt verdict per AI bot, with the deciding rule | Yes, 22 user-agents, RFC 9309 matching | Usually yes, often fewer bots |
| Search vs. training bots shown separately | Yes | Sometimes |
| Live fetch as AI bots (catches Cloudflare/WAF blocks and challenge pages) | Yes: OAI-SearchBot, GPTBot, ClaudeBot, PerplexityBot vs. a browser baseline | Rarely |
| How much text needs JavaScript | Yes, raw HTML vs. rendered copy | Rarely |
| Test a specific path or a draft robots.txt | Yes | Sometimes |
| Copy-paste fix snippet (allow search / block training) | Yes | Sometimes |
| Daily limit / sign-up | None, no sign-up | Some cap free use (for example, Backlinko’s checker allowed 3 free checks a day when we looked in October 2026) |
Access is only step one. To see whether AI answers actually name your brand, run the AI visibility check below.
Free check vs. done-for-you GEO
Access is a yes/no question and this tool answers it for free. Whether AI answers actually recommend you depends on what they read about your category elsewhere.
Free with this tool
- A valid llms.txt built from your sitemap
- A robots.txt check against 22 AI user-agents
- A live fetch as 4 AI bots to catch CDN/WAF blocks and JavaScript-only content
- A copy-paste fix for blocked AI search bots
- Format validation you can rerun any time
What our GEO service adds
- Which buyer questions in your category trigger AI answers, and who gets cited today
- Why competitors are recommended instead of you (sources, pages, mentions)
- Page rewrites and new comparison / FAQ pages built to be quoted
- Ongoing tracking across ChatGPT, Perplexity, Gemini and Google AI Overviews
- Disclosed community work on Reddit and forums that AI answers rely on
Bing site scan: is your site in the index Copilot and ChatGPT search use?
A Bing site scan is a site:yourdomain.com search on Bing that shows which of your pages Bing has indexed. It matters for AI visibility because Microsoft Copilot answers from Bing’s index and ChatGPT search draws heavily on Bing results. If a page is missing from Bing, those engines are unlikely to cite it, however good it is.
- Run it: the checker above links a Bing
site:search for your domain in the report. - Few or no results: confirm Bingbot is allowed in robots.txt (the checker tests it), then submit your sitemap in Bing Webmaster Tools.
- New or updated pages: push them with IndexNow so Bing recrawls within hours instead of weeks.
FAQ
What is the user-agent for ChatGPT’s crawler?
OpenAI uses three tokens: OAI-SearchBot (indexes pages for ChatGPT search), ChatGPT-User (fetches a page when a user asks about it) and GPTBot (training). To be cited in ChatGPT search, allow OAI-SearchBot and ChatGPT-User; GPTBot can be decided separately.
Which AI crawlers should I block?
Most brands that want AI recommendations should allow all AI search and user-fetch bots and keep private paths disallowed. Blocking training bots such as GPTBot, ClaudeBot, Google-Extended or CCBot is a business decision; it does not remove you from AI search answers.
How do I run a Bing site scan?
Search site:yourdomain.com on Bing, or run the checker above, which links that search for you. The result count shows roughly how many of your pages Bing has indexed. Low counts usually mean a robots.txt or firewall block on Bingbot, or no sitemap submitted in Bing Webmaster Tools.
How do I check if my site blocks GPTBot or ClaudeBot?
Enter your domain in the checker above. It fetches your live robots.txt, applies the same matching rules crawlers use, and shows Allowed or Blocked for GPTBot, ClaudeBot and 20 other AI user-agents, with the exact rule that decided it.
Does blocking GPTBot remove my site from ChatGPT?
No. GPTBot collects training data. ChatGPT search results and citations rely on OAI-SearchBot, and ChatGPT-User fetches pages users ask about. Block GPTBot if you want; keep OAI-SearchBot allowed if you want to be cited.
Will allowing AI crawlers hurt my Google rankings?
No. Allowing or blocking AI crawlers has no effect on Googlebot. Google-Extended only controls Gemini training and grounding use; it is not used for Search ranking.
My robots.txt allows everything but AI tools still cannot read my site. Why?
The most common causes are CDN or firewall bot blocking (for example Cloudflare’s AI crawler controls), login or geo walls, and content rendered only by JavaScript. robots.txt cannot override any of these. Leave the live access test switched on: it fetches your page as four AI bots and shows which of these is happening.
Is this AI crawler test free?
Yes, with no sign-up. The tool runs in your browser and fetches robots.txt and your page through the public Jina Reader service; MaxGrowth does not store the domains you test.
Do AI crawlers actually obey robots.txt?
The major operators (OpenAI, Anthropic, Google, Microsoft, Perplexity, Apple) document that their crawlers honour robots.txt, though user-triggered fetchers may be treated differently. Unknown scrapers may not, which is what CDN-level controls are for.
Crawlers can reach you. Now find out whether AI answers recommend you.
Get a free GEO audit