GPTBot and AI crawlers: what to allow in robots.txt

robots.txt is a small public file at the top of a website that works like a sign on the front gate. It tells the automatic page readers from ChatGPT, Perplexity, Google and other AI tools which pages they may visit. One wrong line can stop a search tool reading the page you want customers to find, so it’s worth checking yours if you have one.

These page readers are called crawlers. When a customer asks ChatGPT for a builder, the answer can draw on pages it looked up for search or on what it learned in training. Those are different jobs. Training is when an AI learns from a large body of text before it’s released.

This guide shows you how to let the search tools in, choose separately about training and check that your changes worked. Our AI SEO agency can help if you find an access problem. For the wider work of improving how AI answers describe your business, see what is GEO.

Here’s an example. Say you run a landscaping business in Penrith. A while back, someone helping with your website added a block for AI bots to robots.txt because you didn’t want your writing used for training. The block they copied also turned away ChatGPT’s search reader, called OAI-SearchBot. Now a homeowner asks ChatGPT for a landscaper in Penrith. The search reader arrives at your site and finds the sign telling it to stay out. It leaves without reading a page, so your site isn’t shown in ChatGPT’s search answers. The block was meant to stop training but it switched off search as well.

What does this mean for your business?

For most local businesses the first job is making sure the search readers can get in. Training is a separate choice about how your writing is used. Here’s the order we’d work in.

1. Open yourdomain.com.au/robots.txt in a browser, using the exact address customers use and including www if your site has it.
2. Look for blocks on the search readers, which are OAI-SearchBot, PerplexityBot, Claude-SearchBot and Googlebot. A block on any of them can stop that tool reading your pages.
3. Make training its own decision. GPTBot and ClaudeBot collect pages for possible training and a block on them is only a preference for the future.
4. Keep private pages behind a login, because robots.txt is a sign on the gate and not a lock.
5. After any change, reload the file and give the companies time to catch up. Then check your server records to see who has actually visited.

What is GPTBot and does allowing it help ChatGPT search?

OpenAI doesn’t say that allowing GPTBot helps you appear in ChatGPT search. It ties GPTBot to training and a different reader to search. GPTBot is an automatic page reader that collects material which may train OpenAI’s models. Disallowing it signals that your content shouldn’t be used for training.¹

ChatGPT search uses OAI-SearchBot instead. OpenAI says it surfaces websites in ChatGPT’s search features. A site that opts out of OAI-SearchBot won’t be shown in ChatGPT search answers, though it may still appear as a link people open directly.¹ OpenAI states that each setting is independent of the others.¹

Three words to know. A crawler is software that requests web pages automatically. robots.txt is a public file at the top of your website that tells cooperating crawlers which pages they may visit.⁷ A user-agent token is the crawler name you write in those rules. The file doesn’t log anyone in and it doesn’t keep private information private.

Another file you may hear about, llms.txt, doesn’t control access either. Our llms.txt guide covers what it’s for.

Letting the search reader in gives ChatGPT access to pages it may use but it doesn’t guarantee a mention.¹ Our ChatGPT SEO guide covers the wider work. The LLM SEO guide explains how an assistant can use what it learned or find pages while it answers.

Which AI crawlers do which jobs?

The companies divide the work differently, so match the name in your robots.txt file or server records to the job below. The crawlers in this guide fall into three groups.

• Search readers (OAI-SearchBot, PerplexityBot, Claude-SearchBot and Googlebot) find pages to show in answers.
• Training collectors (GPTBot and ClaudeBot) gather pages that may be used to train a model.
• On-request visitors (ChatGPT-User, Perplexity-User and Claude-User) open a page because one person asked an assistant to.

Google-Extended sits slightly apart. It’s a rule name that controls specified Gemini training and a use called grounding, which means using retrieved content to support an answer. Check that extra use before you treat it as a training-only switch.¹⁴⁵²

Should a small business allow or block AI crawlers?

Most local service businesses want customers to find them in AI search, so allowing the search bots is the sensible default. Blocking training bots is a separate choice about how your content is used.

Anthropic describes a ClaudeBot block as a signal that the site’s future materials should be excluded from training datasets.⁵ None of the pages we read says a block removes content already collected, so treat it as a preference for the future.

Blocking a search bot is different. OpenAI and Anthropic both say it can take your site out of their search results or reduce your visibility.¹⁵

How do you allow search access and block training?

Write a separate group of rules for each bot name. Allow the search bots and disallow the training ones, then read the whole file before you publish it. Treat the example below as a set of choices to review and not as a file to paste over your own.

Each group starts with a User-agent line that names the crawler. An Allow: / line lets it visit the whole site and a Disallow: / line asks it to stay out. Lines starting with # are notes for people.

# OpenAI: allow ChatGPT search, decline training
User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

# Anthropic: allow search and user requests, decline training
User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: ClaudeBot
Disallow: /

# Perplexity: allow search
User-agent: PerplexityBot
Allow: /

# Google: decline Gemini training and grounding, leave Googlebot alone
User-agent: Google-Extended
Disallow: /

Google explains how its page readers pick which group applies. They use the group that matches their name most closely and they don’t combine it with the general * group.⁶ So when you add a group for one crawler, copy across any blocks you already have, because a new group doesn’t inherit them. Those blocks don’t replace a login for confidential content.

Never replace your whole file with the example. Keep the existing blocks for admin or staging areas. Use a login or password to keep confidential material private.⁷ Anthropic supports a setting called Crawl-delay that spaces out requests. It also says each subdomain needs its own rules.⁵

The picture below shows four of those groups as separate switches. ChatGPT search and Perplexity search stay on, while OpenAI training and Gemini reuse go off. Each group stands alone, so changing one leaves the others as they were.

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Disallow: /

What does Google-Extended control?

Google-Extended controls training for future Gemini models. It also controls specified uses of retrieved content to support answers, called grounding, in Gemini Apps and Vertex AI. It doesn’t control inclusion or ranking in Google Search.² Decide these reuse permissions separately from whether customers can find your pages in Search.

You won’t see Google-Extended as a visitor in your server records. It has no separate name in the requests your server receives, because Google uses its existing page readers. It exists only as a rule name in robots.txt.²

Google’s separate Search generative AI setting controls participation in AI Overviews, AI Mode and the generative features in Discover. It doesn’t control model training.³ Blocking Googlebot can stop Google reading your pages. Even so, robots.txt doesn’t reliably remove those pages’ addresses from Search.⁷ Our AI Mode and AI Overviews guides explain the participation setting.

Do visits started by a person follow the same rules?

Not always. OpenAI says ChatGPT-User actions are started by a person, so robots.txt rules may not apply.¹ Perplexity says Perplexity-User generally ignores robots.txt for the same reason.⁴ Anthropic documents robots.txt compliance for its bots.⁵

robots.txt isn’t a security boundary. Google says a blocked page’s address can still appear in Search.⁷ So it works as a sign and not a lock. Anything confidential needs a login. When a person asks an assistant to open a public page, the visit may follow different rules from an automatic crawler, depending on the company.¹⁴⁵

Why might an allowed crawler still not reach your page?

Your host or its security filters can block requests even when robots.txt allows them. A content delivery network, which is a network that delivers saved copies of your website files, may apply its own filters too. Ask your host to check whether a legitimate bot receives the page or an error.

Three checks narrow it down.

1. Use the right address, because rules apply only to the exact host, protocol and port that serve the file. A subdomain, such as shop.yourdomain.com.au, needs its own robots.txt.⁶
2. Check that the file loads, because a broken file changes what Google does. If robots.txt returns a 404, which is the “page not found” error, Google treats the site as having no restrictions. If the server returns an error instead, Google first stops its automatic page reading for 12 hours while it retries. It can then use the last good copy for up to 30 days. If none is available, it assumes no restrictions. Google’s documentation explains the later steps.⁶
3. Check who is really visiting. Anyone can copy a bot’s name, so compare the visitor’s network address with the company’s published list. OpenAI and Perplexity provide those lists. Perplexity advises checking both the name and the address.¹⁴ Anthropic also publishes addresses but warns that blocking them can stop its bots reading robots.txt. That means an address block may not reliably tell Anthropic you want to opt out of training.⁵

If you use a service such as Cloudflare, Perplexity says to set its bots to Allow so their requests bypass other security rules. It also says to refresh the lists of network addresses automatically.⁴ Check your own provider’s documentation before copying that setting.

How do you check that a robots.txt change worked?

Changing the file isn’t the last step. Open it, read the rule that applies, give the companies’ systems time to catch up and then look at your records.

1. Open the live file by visiting yourdomain.com.au/robots.txt on the exact address customers use, including www if you use it.
2. Find the group for each bot. Then confirm it’s the one you meant and that your rules for private areas are still there.
3. Allow time for updates, because OpenAI gives about 24 hours for its search systems to adjust to permission changes. Perplexity says its systems may take up to 24 hours. These statements are not universal deadlines for every bot and they don’t prove that a block has taken effect.¹⁴
4. Check your server records by looking for the bot names, then compare the addresses with each company’s published lists.
5. Test the page by making sure the public version loads without a login or a challenge screen.

A bot reaching your page, a search engine holding it and an answer citing it are three separate outcomes. The first doesn’t prove the second or the third.

Kavera note: In the Sydney trades AI search study we reported blocked, failed and no-answer checks separately from a business being absent. The same idea applies here. If an assistant can’t read your page, that’s an access problem and says nothing about how good the page is.

Kavera note: In two checks about two months apart, our client SHORE went from absent to first in one Sydney builder shortlist. The ChatGPT SEO guide records that one question. The comparison doesn’t establish whether any individual website change, including a change to robots.txt, caused the result.

What does each company say about its crawlers?

This table is the reference version of the three groups above. Don’t flatten the rows into one rule, because the companies treat these bots differently. Perplexity doesn’t describe either of its listed tools as collecting content to train AI foundation models, the large models behind AI assistants.⁴ OpenAI and Perplexity say the fetchers a person starts may or generally do ignore robots.txt. Anthropic says its bots honour robots.txt directives.¹⁴⁵ Our Perplexity SEO guide covers how that engine picks sources.

What should you do first?

Open your website’s robots.txt file and check for accidental blocks on the search bots. Make your training choice separately. Then check that the search tools you meant to allow can reach your public pages.

If you want help, our AI SEO agency can run the access check with you. Our SEO audit is free. It reviews your technical setup, keywords, competitors and Google Business Profile, then gives you a prioritised roadmap. Implementation is separate unless retained. Monthly plans start at Suburb SEO, $890/month + GST. See the SEO packages for scope and terms. No SEO package guarantees rankings, leads or revenue.