What Is AI Training Data, and Is Your Website In It?

You have heard that ChatGPT "learned from the internet". Your website is on the internet. So it is a fair thing to wonder: did it read your site, and is that why it does or does not mention you?

What is AI training data? It is the text a model was built from before anyone could use it. Whether your website is in it is a question nobody outside the AI company can fully answer, and it matters less to your business than it sounds.

Below is what training data is, who collects it and under what names, what you can and cannot check about your own site, whether being in it gets you recommended, and how to decide about opting out.

What is AI training data?

AI training data is the large body of text a model was built from before it was released. For the assistants your buyers use, much of that text is public web pages, collected in bulk by automated visitors. The model learns patterns from it. It does not keep a searchable copy, and it does not go back for updates.

A plain way to hold it: training data is a reading list that closed on a certain date. The model read everything on the list once, formed a general picture of how the world is described, and was then released. What it reads on the list is fixed from that day. That date is called the knowledge cutoff, and it has its own post: what a knowledge cutoff is.

Two things follow from that, and they are the whole reason this matters to a coach or a consultant.

• It is old. Anything you changed after the list closed is not in it. New prices, a new specialty, a new city.
• It is blurry. A model that read one page among billions does not hold that page word for word. It holds a general impression, and for a small business the impression can be faint or missing.

Training data is also only one of the ways an assistant comes to say something. The other is looking things up live, at the moment a person asks. That second route is where most of the useful work is, and we will come back to it.

Where it comes from, and who collects it

Most of it is collected by automated visitors, usually called crawlers, that fetch public pages one after another. Each AI company names its collector, and a site owner can address that name in a file called robots.txt. Four names cover most of what a small business needs to know. The table lists them with what each owner says.

The sources, so you can read them yourself. OpenAI's page on its crawlers lists GPTBot separately from OAI-SearchBot and says each setting is independent of the others. Anthropic's help page on whether it crawls the web describes three visitors with three jobs. Google's list of its common crawlers explains Google-Extended and says it is not a ranking signal in Search.

Common Crawl is the odd one, and the most interesting for this question. Its home page describes a free, open repository of web crawl data that can be used by anyone, holding over 300 billion pages spanning 15 years, with 3 to 5 billion new pages added each month. "Anyone" includes companies that build AI models. The home page does not say which ones, and we have not verified any company's use of it.

Public web pages are not the only thing a model can be built from. They are the part a small business website can end up in, so this post stays there.

Is your website in it?

Possibly, and you cannot confirm it. We know of no public list of the pages any one model learned from, so nobody outside the company can tell you for certain. What you can do is check three things that make it more or less likely. Each takes a few minutes.

1. Read your robots.txt file. Type your web address followed by /robots.txt into a browser. If you see a collector's name with "Disallow: /" under it, you have asked that collector to stay out. If the file says nothing about them, you have not.
2. Look in your visitor logs. If your host gives you raw access logs, search them for GPTBot, ClaudeBot and CCBot. A visit means the page was fetched. It does not tell you what was done with it afterward. How to read those lines is in what GPTBot is.
3. Search the Common Crawl index. This is the one public archive that lets you look up a web address. The figure shows what the lookup can and cannot tell you.

THE ONE PLACE YOU CAN LOOKSOURCE: COMMONCRAWL.ORG, READ OCT 10, 2026

Common Crawl index, one crawl selected

yourdomain.com/*

• Rows come back

  Those pages were captured in that crawl.

• No rows

  Nothing from your site is in that crawl. Pick another crawl and look again.

What neither result tells you

• Which AI companies used that crawl, if any
• Whether your pages were kept or dropped along the way
• What an assistant says about you today

A copy in the archive is a possibility. It is not a place in a model.

AnswerHalo answerhalo.com

CLICK TO ENLARGE

The lookup lives at index.commoncrawl.org. You choose a crawl from a list, type your web address with /* on the end, and it returns the pages it captured. It is a tool built for researchers and it looks like one. The results come back as lines of raw data, one per page.

Be clear about what a result means. Rows mean a copy of those pages sat in that one crawl. No rows means that crawl missed you. Neither says a model was trained on that crawl, or that your pages made it through whatever sorting happened next. And Common Crawl is one archive. The AI companies also run collectors of their own.

So the honest answer to "is my website in it" has three parts. If your site is public, has been online for more than a few months and does not block the collectors, it has probably been fetched by at least one of them. Whether it was used, nobody will confirm. And if it was, the model has an impression of it, not a copy.

Does being in it get you recommended?

No. Being collected is not the same as being remembered, and being remembered is not the same as being recommended. For a small business, what gets you named is mostly what an assistant finds when it looks things up at the moment a buyer asks.

Here is why training data does so little for a solo practice. A model read your services page once, in a pile of billions of pages. A household name appears in that pile thousands of times, described the same way each time, and leaves a strong impression. A coach in Denver with one website appears a handful of times. The impression is thin. That is our reasoning from how these systems are described, not a measurement we have run.

The live route is different in three ways that all favor you:

• It is current. A page you fixed last week can be read this week. OpenAI's crawler page says search results can take about 24 hours to adjust after a site changes its robots.txt file, which gives a sense of the pace.
• It is specific. The assistant reads the actual words on the page, so a clear sentence can be repeated closely.
• It is in your hands. You cannot edit what a model learned last year. You can edit your page this afternoon.

The two routes move at very different speeds, and the detail of that is in how often ChatGPT updates what it knows about your business. The practical upshot is short. If an assistant describes an old version of you, the old picture is probably coming from training. The fix is to make the current facts easy to find and consistent everywhere, which is covered in how to keep one page current so AI quotes it right.

One honest limit. Not every answer involves a live lookup. Some questions an assistant answers from what it learned, with no search at all, and in those answers training data is all there is. You cannot tell from the outside which kind you will get every time. That is one more reason to read real answers and not reason about the archive.

Should you opt out?

That is your call, and it is a reasonable one either way. Opting out of training keeps your future pages off the reading list. It does not make you easier or harder to find, as long as you block only the training collectors and leave the search ones alone.

Reasons an owner opts out: the site holds paid course material, original research or writing they sell, and they do not want it used without payment. Reasons an owner stays in: the site is marketing, written to be read and repeated, and a model that has seen it is no loss.

If you do opt out, three cautions:

1. Block by name, one at a time. GPTBot and ClaudeBot are the training collectors. OAI-SearchBot, Claude-SearchBot, ChatGPT-User and Claude-User are how you get into answers. A blanket "block all AI" setting usually catches both kinds.
2. Read the Google line twice. Google-Extended covers training and also Gemini looking your pages up. It is not a training-only switch, so blocking it costs more than the other two.
3. Expect it to apply forward. Anthropic's page speaks of a site's future material. A block now does not pull back what an earlier model already read.

Whichever way you go, the question that pays is not "am I in the archive". It is "what does the assistant say when someone asks for a business like mine". You can see that directly. The free check asks ChatGPT 25 real buyer questions from your niche at $0 and emails a plain-English readout with your score, who got named in your place, and the one fix to start with. It runs as soon as you ask for it, and the report lands within the hour between 7am and 5pm Central, or by 10am the next morning outside those hours.

What the free check does not do: it covers ChatGPT only, it does not open your robots.txt file, and it cannot tell you what any model was trained on. Nobody can. It tells you what was said, which is the part a buyer sees.