Key Takeaways
- Check four places, not one. A block can sit in robots.txt, in a security service in front of your site, in a tag on the page, or in how the page loads. Robots.txt is only the first.
- Never trust robots.txt alone. A security rule can refuse an AI crawler before it reads a single page, and nothing in robots.txt will show it.
- Ask your web person what was switched on. The most common blocks were never written by hand. They are a setting someone flipped in a website builder, host or security service.
- Read your visitor log for the final word. A line naming OAI-SearchBot with a 403 beside it is a refusal. That is the one piece of evidence that does not depend on guessing.
- Give it a day after you fix it. OpenAI says ChatGPT search can take about 24 hours to adjust to a robots.txt change, so check again tomorrow, not in five minutes.
Your robots.txt file looks fine. Your web person says nothing is blocked. And ChatGPT still never mentions your business when a buyer asks for someone like you.
So how do you check if your website blocks AI crawlers? Look in four places, not one, because robots.txt is only where the most visible block lives. The others sit in security settings, page tags and the way a page loads, and they do not show up in that file at all.
Below are the four places, an example of a site that says yes and no at the same time, a 15-minute check you can do yourself, and what to do if you find something. The example site and its results are made up.
How to check if your website blocks AI crawlers
Check robots.txt, then your security service, then your page tags, then whether your text is readable without a login. Finish by searching your visitor log for the crawler names, which shows what actually happened rather than what was meant to.
An AI crawler is a program an AI company sends to read public web pages. Each company runs more than one, for different jobs. The ones that matter if you want to be recommended are the search crawlers. OpenAI's crawler documentation says OAI-SearchBot "is used to surface websites in search results in ChatGPT's search features." If that crawler cannot read your pages, ChatGPT search has nothing of yours to quote.
Which crawlers follow robots.txt, and which ones you might choose to block on purpose, is its own subject, covered in do AI crawlers obey robots.txt. This post is about finding blocks you did not choose.
The four places a block can live
A block can live in robots.txt, in a security service in front of your site, in a tag on the page, or in how the page is built. Each one is checked differently, and each one is usually switched on by a different person.
| Where the block lives | What it looks like | How to check | Who usually set it |
|---|---|---|---|
| Robots.txt | A crawler name, or User-agent: *, followed by Disallow: / | Open yoursite.com/robots.txt in a browser and read it | A web person, a plugin, or the website builder |
| Security service or firewall | The crawler is refused with an error such as 403 before it reads anything | Look at the bot settings in the service, then the visitor log | Whoever set up the hosting or security service |
| Website builder or host setting | A switch labeled something like "block AI crawlers" | Search the builder's settings for "crawler" or "AI" | The site owner, often without knowing what it changes |
| Page tags | A noindex tag in the page code, or the same rule sent as an X-Robots-Tag header | View the page source and search for "noindex" | A plugin, a staging setting left on, or a web person |
| How the page is built | Your words sit behind a login, or only appear after the page's scripts run | View the page source and search for a sentence from the page | The website builder or the page design |
Page tags work differently from the rest. Google's guide to the robots meta tag says noindex means "Do not show this page, media, or resource in search results." That keeps a page out of Google, and so out of the AI Overviews built on Google's index. The same guide adds a trap: if robots.txt blocks a page from being crawled, Google cannot see its noindex rule at all.
Why robots.txt can say yes while your site says no
Robots.txt can say yes while your site says no because a security service answers first. A crawler that is refused at the door never gets as far as reading robots.txt, so the file can be wide open and still not matter.
Security services are the usual source. Cloudflare, a security and speed service that sits in front of many websites, documents AI crawler rules with three choices: "Block (on all pages)", "Block on pages with ads", and "Allow (do not block)". Its rules sort AI crawlers by purpose, such as training and search, and a block set in that account is invisible from robots.txt.
- Your own browserA person Robots.txt says: Allowed Security rule: Let through What came back: Page sent (200)
- GooglebotGoogle Search Robots.txt says: Allowed Security rule: Let through What came back: Page sent (200)
- OAI-SearchBotChatGPT search Robots.txt says: Allowed Security rule: Refused What came back: Refused (403)
- PerplexityBotPerplexity search Robots.txt says: Allowed Security rule: Refused What came back: Refused (403)
- Claude-SearchBotClaude search Robots.txt says: Allowed Security rule: Refused What came back: Refused (403)
Robots.txt allows all five visitors. Three were refused by a security rule before they read a single page.
The site in the example is illustrative. The pattern is real: the owner checks robots.txt, sees nothing blocked, and stops looking. The only place the refusal shows up is the visitor log, as a crawler name with a 403 beside it.
Notice also what the owner sees in their own browser: the page, every time. A block on AI crawlers is invisible from the front of the site, which is why it can sit there for months. It is one of the quieter reasons covered in why ChatGPT cannot find your business.
Check your site in 15 minutes, step by step
Check your site for AI crawler blocks in five steps, in this order, because each one rules out a place the next one cannot see. Most sites need about 15 minutes, plus a message to whoever set up the hosting.
- Read robots.txt. Type your address followed by /robots.txt. Look for OAI-SearchBot, Claude-SearchBot, PerplexityBot, and any group headed User-agent: * with Disallow: / under it. Either blocks the crawler from the whole site.
- Check the security service. If your site sits behind Cloudflare or a similar service, open its bot settings and look for anything that blocks AI crawlers. If you do not know whether you have one, ask whoever set up your domain.
- Check the builder or host. Search your website builder's or host's settings for "crawler" or "AI". Some offer a single switch that blocks AI crawlers across the site.
- Check your key pages. On your home page and main services page, view the page source (Ctrl+U in most browsers). Search it for "noindex", then for one sentence you can see on the page. If the sentence is not in the source, a crawler that does not run the page's scripts may not see it either.
- Read the visitor log. Ask your host for the raw visitor log and search it for OAI-SearchBot. A line with 200 means the page was sent. A line with 403 means it was refused. No lines at all over several weeks is worth asking about too.
A word on shortcuts. Some guides suggest testing from your own computer while pretending to be a crawler. Security services often check where a request comes from, not just the name it gives, so a test like that can be refused when the real crawler is let in, or the reverse. OpenAI publishes the addresses its search crawler uses, which is how a security service tells the real one from an impostor. The visitor log records the real visits, which is why it gets the final word.
Pasting your link into ChatGPT is not a shortcut either. That request comes from ChatGPT-User, which OpenAI lists separately from the search crawler, so a page that opens when you paste it can still be closed to search.
What to do when you find a block, and what this will not tell you
When you find a block, change it where it was made, then wait a day and check the visitor log again. Editing robots.txt will not undo a rule set in a security service, and the reverse is also true.
Before removing anything, ask why it is there. A rule refusing training crawlers such as GPTBot may be deliberate, and it does not stop search crawlers. A rule refusing everything with "AI" in its purpose is usually an accident. OpenAI says it can take about 24 hours after a robots.txt update for ChatGPT search to adjust, so give a fix a day before judging it.
The damaging admission: an open door does not get you named. This check tells you whether AI crawlers can read your site. It cannot tell you whether they will recommend you, which depends on whether your pages answer the questions buyers ask, and on what other sites say about you. A file like llms.txt does not change that either; see what llms.txt is and whether it helps.
To see whether an open site is turning into being named, start with the free check. It asks ChatGPT 25 real buyer questions from your niche at $0 and emails a plain-English readout: your score, who got named in your place, and the one fix to start with. It runs as soon as you ask for it, and the report lands within the hour between 7am and 5pm Central, or by 10am the next morning outside those hours.
Questions we hear the most
How do I check if my website blocks AI crawlers?
Check four places: your robots.txt file, any security or firewall service in front of your site, the tags on your key pages, and whether your page text is visible without logging in. Then search your visitor log for the crawler names to confirm what actually happened.
Can robots.txt allow AI crawlers while my site still blocks them?
Yes. Robots.txt is only a request that crawlers read. A security service or firewall rule can refuse an AI crawler at the door, so it never reaches robots.txt or any page, and the file itself will look perfectly open.
Does Cloudflare block AI crawlers?
It can, if the setting is on. Cloudflare offers AI crawler rules with three choices: block on all pages, block only on pages with ads, or do not block. Look under the bot settings in your Cloudflare account to see which one your site uses.
If ChatGPT opens my page when I paste the link, can its search crawler read my site?
Not necessarily. A pasted link is fetched by ChatGPT-User, which OpenAI lists separately from OAI-SearchBot, the crawler that puts sites in ChatGPT search. A security rule can treat the two differently, so one working does not prove the other does.
How long after I remove a block will ChatGPT notice?
OpenAI says it can take about 24 hours after a robots.txt update for its search systems to adjust. Being readable again is the start, not the finish: being named still depends on what your pages say.
Should I unblock every AI crawler?
No. If you want to be recommended, keep the search crawlers open, such as OAI-SearchBot, Claude-SearchBot and PerplexityBot. Training crawlers like GPTBot and ClaudeBot are a separate choice you can make either way.
Does the AnswerHalo free check test whether my site blocks AI crawlers?
No. The free check does not touch your site. It asks ChatGPT 25 real buyer questions from your niche at $0 and emails your score, who got named in your place, and the one fix to start with.
What does the $500 AnswerHalo audit add?
The audit is $500 once and runs 125 checks across ChatGPT, Claude, Gemini, Perplexity and Google AI Overviews, keeping every answer so you can read it, and lands by this time tomorrow. The free check covers ChatGPT only, with 25 questions.
Your buyers are already asking. Find out what AI tells them.
The free check asks ChatGPT 25 real buyer questions about your niche and sends you the report within the hour, at $0. If it shows you are already getting named everywhere, we will say so in plain words.
Get my free check →