Key Takeaways
- Stop picturing Claude clicking around your site. It does not wander from link to link. It either searches, opens one specific page it has been pointed to, or answers from what it already learned in training.
- Get found by search before you worry about being read. Claude can only open an address that has already appeared in the conversation, from the buyer or from a search result. A page nobody names and no search surfaces is never opened.
- Put your key facts in plain text on the page. Anthropic says its page-reading tool does not support sites that are built by JavaScript in the browser. Text that only appears after a script runs may not be read at all.
- Decide on each of Claude’s three crawlers separately. One gathers training data, one builds search results and one fetches pages when a person asks. Blocking the last two can make you harder to find in Claude’s answers.
- Check for sources before you read the answer. When Claude searches, it shows where it looked. An answer about you with no sources attached came from memory, and publishing today will not change that particular answer.
Ask a business owner whether Claude browses the web and you usually get one of two pictures. Either it reads everything, all the time, or it reads nothing and only knows what it was trained on.
Neither is right, and the difference matters. Whether Claude browses the web decides whether a change to your website can show up in its answers this month or has to wait for a future model.
Here is what Claude actually does with the web, the one rule about addresses that most owners never hear, four ways Claude can know about your business side by side, and what it means for your site.
Does Claude browse the web?
Claude can search the web and open specific pages, but it does not browse the way a person does. It never wanders from link to link. It looks things up when a question needs it, and otherwise it answers from what it learned in training.
So "does Claude browse the web" has a three-part answer. Sometimes it searches. Sometimes it opens one page it has been pointed at. And a great deal of the time it does neither, because the question did not call for it or the feature was not switched on where the buyer asked.
For your business, that means there are two different ways to be in an answer. One is being part of what Claude already knows. The other is being findable and readable at the moment it goes looking. They are fixed on very different timescales.
The two ways Claude reaches a page
Claude reaches the live web through two separate tools: one searches, one opens a single page. Anthropic documents both, and the rules for each are different.
Anthropic's developer documentation for the web search tool says Claude searches when a request depends on information that is current, changing or outside its training data, and lists "information about specific organizations, people, or products that might have changed" as one of the triggers. A buyer asking who coaches new managers in Denver is asking exactly that kind of question.
The second tool opens a page. The web fetch tool retrieves the full text of a specific page or PDF, and Anthropic says Claude uses it when a request points at a particular page. It also says Claude does not fetch for general, open-ended questions. "What should I look for in a coach?" gets answered from memory.
One honest limit before going further. These are the tools Anthropic documents for developers. The Claude app a buyer types into is built on them, but Anthropic does not publish every setting the app uses, and web search can be switched off. What follows describes what Claude can do, not a promise about every conversation.
Claude only opens addresses it has been given
Claude can only open a page whose address has already appeared in the conversation. It cannot type in an address it thought of itself. That one rule decides which of your pages it is able to read at all.
Anthropic states it directly: the fetch tool can only open addresses that came from a user's message or from earlier search or fetch results, and it cannot open addresses that appear only in Claude's own output. The rule exists for security, but its side effect is plain. To be read, your page has to be named by the buyer or surfaced by a search first.
-
denvercoach.example/pricing
Buyer’s messageSearch resultClaude’s own replyNot in the chatCan open: The buyer pasted it
-
denvercoach.example/about
Buyer’s messageSearch resultClaude’s own replyNot in the chatCan open: A search turned it up
-
coachlist.example/denver
Buyer’s messageSearch resultClaude’s own replyNot in the chatCan open: A search turned it up
-
denvercoach.example/faq
Buyer’s messageSearch resultClaude’s own replyNot in the chatCannot open: Claude guessed the address
-
denvercoach.example/reviews
Buyer’s messageSearch resultClaude’s own replyNot in the chatCannot open: Nobody mentioned it
Nothing about the five pages differs. The verdict follows the column: an address the buyer typed or a search returned can be opened, and an address Claude guessed, or one nobody mentioned, cannot.
The addresses in that figure are illustrative. The pattern is the point: your reviews page can be the best page on your site and still never be read, because nothing in the conversation ever pointed at it. Search is the door. The page-reading tool only walks through doors search has already opened, or that the buyer opened by pasting a link.
Two more limits sit on top of that rule. Anthropic says the fetch tool does not support websites that are built by JavaScript in the browser, and that it refuses addresses blocked by a site's robots.txt file. A page can be found and still come back empty.
Four ways Claude can know about you
Claude can know about your business in four ways, and only two of them happen while the buyer waits. The table sets them side by side with what each one reads and what can stop it.
| Route | When it happens | What it reads | What can stop it |
|---|---|---|---|
| Training (ClaudeBot) | Months before the question | Pages collected for a future model | Blocking ClaudeBot keeps your future pages out of training |
| Search index (Claude-SearchBot) | Ahead of time, continuously | Pages indexed to improve search results | Blocking Claude-SearchBot, per Anthropic, may reduce your visibility |
| Web search in the chat | While the buyer waits | Search results for the buyer's question | Search switched off, or a question Claude answers from memory |
| Opening one page (Claude-User) | While the buyer waits | The full text of one named page | An address not in the chat, a robots.txt block, a page built by JavaScript |
The crawler names come from Anthropic's help article on how Anthropic crawls the web, which says that disabling Claude-User prevents Claude from retrieving your content in response to a user's question. Read down the last column and most of what stops Claude is a setting on your own site.
What this means for your website
Make your important pages easy to find by search and easy to read as plain text. That covers most of what Claude's web access asks of a business owner.
In practice, three checks do the most work. Open your robots.txt file and make sure Claude-SearchBot and Claude-User are not blocked, unless you mean them to be. View your key pages with JavaScript turned off, or ask your web person to, and confirm your price, location and who you serve are still visible. And put the facts a buyer would ask about on the pages a search is likely to return, rather than on a page nothing links to.
The damaging admission: none of this makes Claude recommend you. Being readable only means that when Claude looks, it can see what you wrote. Whether it names you depends on whether what it finds, on your site and elsewhere, answers the buyer's question better than the alternatives. For how Claude weighs that, see how Claude decides what to recommend.
Many answers will still come from memory, which is why an old description of you can outlive a new website. For how that memory ages, see what a knowledge cutoff is, and for the same split in OpenAI's assistant, see ChatGPT search versus ChatGPT without browsing.
If you want to know whether ChatGPT names you today, start with the free check. It asks ChatGPT 25 real buyer questions from your niche at $0 and emails a plain-English readout: your score, who got named in your place, and the one fix to start with. It runs as soon as you ask for it, and the report lands within the hour between 7am and 5pm Central, or by 10am the next morning outside those hours.
Questions we hear the most
Does Claude browse the web?
Not the way a person does. Claude can search the web and open specific pages when a question needs current information, but it does not click from link to link on its own, and when it does not search it answers from what it learned in training.
Can Claude read my website if someone gives it the link?
Usually, yes. Anthropic documents that Claude can open a page whose address appears in the conversation, including one a person pastes in. It cannot open addresses it simply makes up, and it cannot open a page your robots.txt file blocks.
Can Claude read a website built with JavaScript?
Not reliably with its standard page reader. Anthropic says its web fetch tool does not support websites dynamically rendered with JavaScript, so a page whose words appear only after a script runs in the browser may come back thin or empty.
Should I block Claude’s crawlers?
Only the one you have a reason to block. Anthropic runs three: ClaudeBot collects training data, Claude-SearchBot improves search results, and Claude-User fetches pages when someone asks. Blocking the last two can reduce how visible you are in Claude’s answers.
Does Claude always show where it got its information?
When it searches, yes. Anthropic’s documentation says citations are always enabled for web search. When Claude answers from memory, nothing is attached, because it did not look anything up for that answer.
Is what Claude reads always the latest version of my page?
Not always. Anthropic notes that its page-reading tool caches results, so the content returned may not reflect the latest version at the address. A change you made this morning may not be what Claude reads this afternoon.
Does the free check test Claude?
No. The free check asks ChatGPT 25 real buyer questions from your niche at $0 and shows whether you are named and who is named instead. Claude is one of the five assistants in the $500 audit.
What does the $500 audit show about Claude?
Whether Claude names you, and who it names instead. The audit runs 125 checks across ChatGPT, Claude, Gemini, Perplexity and Google AI Overviews, keeps the answers so you can read them, and lands by this time tomorrow.
Your buyers are already asking. Find out what AI tells them.
The free check asks ChatGPT 25 real buyer questions about your niche and sends you the report within the hour, at $0. If it shows you are already getting named everywhere, we will say so in plain words.
Get my free check →