Free tools · AI Search
AI Crawler Checker
Paste a URL and see which AI bots your site lets in, and which it turns away.
Free AI crawler checker
Before you start
Enter a domain to see which of 26 AI crawlers your robots.txt file lets in, and which it turns away.
For every bot you get the same four things: who runs it, what it feeds, whether it is allowed or blocked, and the exact line in your file that decided it. Then you get a ready made robots.txt you can edit.
Reading the file
Fetching robots.txt.
This takes a few seconds, and longer if the server is slow.
Check stopped
Check the spelling and try again. If the site sits behind a firewall or a CDN filter, it will refuse us the same way it refuses AI crawlers, and that is worth knowing on its own.
Results
Could not verify, so this is not a result
We did not read a rule, so we report no verdict for any bot. That is not the same as zero blocked and it is not the same as all blocked. It is unmeasured, and saying anything else would be a guess.
A site that refuses a plain request like ours is usually sitting behind Cloudflare, Akamai or a similar shield. Most AI crawlers are plain clients too, so they may be hitting the same wall.
robots.txt
-
AI crawlers let in
-
llms.txt
-
How to read the verdicts
Allowed and Partly blocked both mean the bot can reach your home page. Partly blocked means the same rules also shut some paths, an admin or basket folder for example, which is normal and usually on purpose. Only Blocked means the bot is turned away at the front door. Every verdict names the line it came from.
Your robots.txt file is larger than the amount we read, so any rule past that point was not checked.
Build a new robots.txt
Tick the bots you want to let in. Untick the ones you want to keep out.
The boxes start on your current position, so nothing changes until you change it. Any path you already close for everyone is repeated inside each bot's own group. That repeat matters: once a bot has a group of its own it stops reading your catch-all rules, so a bare Allow would quietly reopen the folders you had shut.
Two retired tokens, anthropic-ai and Claude-Web, are left out of the file on purpose. No live crawler answers to them any more, so a rule for either one does nothing.
Your new robots.txt
No file to build
We could not read your current file, so we will not write you a new one.
A generated file replaces everything you have. Without sight of your existing rules we would hand you a file that quietly deletes the ones you rely on. Get the file readable first, then run this again.
What this check does not tell you
- robots.txt is a request, not a lock. Well behaved bots follow it. Badly behaved ones read it and carry on.
- A firewall or CDN can turn a bot away even when your file says come in. That happens at the server and leaves no trace in this file.
- Letting a bot in does not mean it will quote you. That depends on what you have written and who links to you.
- We check the home page only. A rule can open the front door and still shut most of the rooms.
- Nothing here is asked of an AI model. Every verdict comes from a file we fetched and read, so the same site gives the same answer every time.
FAQ
AI Crawler Checker, answered.
Is GPTBot blocked on my site?
Paste your address above and you will see. The tool reads your robots.txt file and shows the exact line that lets GPTBot in or keeps it out, along with the other AI bots. If no line matches it, the bot is allowed.
What is robots.txt?
A plain text file at the root of your site that tells automated visitors where they may go. It is a request, and the big search and AI companies honour it. Block the wrong line and you can vanish from a whole platform without noticing.
Should I block AI crawlers?
Only if you do not want to be quoted. Blocking GPTBot keeps your pages out of ChatGPT's answers, and that is credit going to whoever left the door open. Publishers who sell access to their archive are the exception.
Which AI bots does this check?
The crawlers used by OpenAI, Anthropic, Perplexity, Google, Apple, Meta, ByteDance and Common Crawl, plus the smaller ones. Each is listed with the exact rule line that decides it, so you can see why it was allowed or blocked.
Where this stops being a calculator
Open the door and the bots arrive. What they find is the part we work on: pages that answer a real question in the first paragraph, in words a machine can lift cleanly.
Two marketers. Your whole account.
Book a call and tell us what you want to grow, or get the free website report first.