How to Configure robots.txt for AI Search and Model Training

What each AI crawler does

A bot's purpose determines the effect of allowing or blocking it. Use the search bot's rules for search access and the training bot's rules for training use.

Bot or control	Purpose	When to allow or block it
OAI-SearchBot	Inclusion in ChatGPT search	Allow access to pages intended for search
GPTBot	Collecting content that may be used to train OpenAI models	Block access to decline training use
PerplexityBot	Inclusion in Perplexity search	Allow access to pages intended for search
Googlebot	Crawling for Google Search	Allow access to pages intended for Search and AI Overviews
Google-Extended	Training and answer grounding in specified Gemini products	Block it to decline those uses

OpenAI documents the search and training roles shown above. Perplexity states that PerplexityBot collects content for search rather than foundation model training. (OpenAI: Overview of OpenAI crawlers, Perplexity: Crawlers)

Googlebot controls crawl access for AI Overviews and AI Mode. Google-Extended controls content use in Gemini Apps and specified Vertex AI features. Blocking Google-Extended leaves Google Search inclusion and ranking unaffected. (Google Search Central: AI features and your website, Google: Google-Extended)

How to configure robots.txt

Write the bot names and allowed or blocked paths in robots.txt, then publish it at the site root. If a file already exists, preserve its settings when adding the new rules.

Specify the bot and page paths

The three directives below identify the bot and control access. A path is the part of a URL after the domain, such as /blog/ or /admin/.

Directive	Meaning	Example
User-agent	Bot to which the rules apply	User-agent: OAI-SearchBot
Allow	Path the bot can retrieve	Allow: /blog/
Disallow	Path the bot should avoid	Disallow: /admin/

User-agent: * applies when the crawler has no matching named group. / covers the whole site, so Disallow: / blocks crawling across it. (IETF: Robots Exclusion Protocol)

Allow search bots and block the training bot

This example lets OAI-SearchBot and PerplexityBot retrieve public articles while blocking GPTBot across the site. It excludes /admin/ and /account/ from general crawling and from both search bots.

User-agent: *
Disallow: /admin/
Disallow: /account/

User-agent: OAI-SearchBot
Allow: /
Disallow: /admin/
Disallow: /account/

User-agent: PerplexityBot
Allow: /
Disallow: /admin/
Disallow: /account/

User-agent: GPTBot
Disallow: /

Replace /admin/ and /account/ with the paths your site excludes. Keep existing exclusions and Sitemap entries. Articles under /blog/ remain accessible to both search bots in this example.

Include exclusions in each bot's named group. Wildcard exclusions are not inherited when a matching named group applies. That is why /admin/ and /account/ appear in both search bot groups above. (IETF: User-agent rules)

Publish robots.txt at the site root

Publish the file at the site root, for example https://example.com/robots.txt. A file at https://example.com/blog/robots.txt cannot serve as the site's robots.txt. Rules apply to the host and protocol where the file is published, so check other subdomains too. (Google: Create and submit a robots.txt file)

Open the published robots.txt URL to confirm the bot names and paths are present. Correct any rules that block articles intended for search.

What Google-Extended controls

Google-Extended covers training and answer grounding. Grounding means supplying information from the Google Search index to help a model answer a question. The control applies to training future Gemini models and grounding in Gemini Apps and specified Vertex AI features. (Google: Google-Extended)

To decline those uses, add:

User-agent: Google-Extended
Disallow: /

Google-Extended specifies how content collected by Google's existing crawlers may be used. Those crawlers use their existing HTTP user-agent strings, so access logs will show those names rather than a dedicated Google-Extended crawler. (Google: Google-Extended)

For content controls within AI Overviews and AI Mode, use nosnippet, data-nosnippet, or max-snippet. These directives restrict Search snippets and the content available for generating answers. A page must be indexed and eligible for a Search snippet to qualify as a supporting link in Google's AI features. (Google Search Central: AI features and your website)

For limiting selected text, read our guide to data-nosnippet.

Check the WAF and page response after changes

A web application firewall, or WAF, detects and blocks suspicious requests. It or the server can deny a bot access even when robots.txt allows crawling. Perplexity recommends checking both the bot name and its official IP address when configuring access. (Perplexity: WAF configuration)

After publishing the rules, check the target page for these issues.

Check	Action if there is a problem
Bot names and excluded paths in robots.txt	Correct rules that block public articles
WAF block logs	Identify rules denying a bot request from an official IP address
Page HTTP response	Investigate errors or unexpected login pages
Retrieved HTML	Investigate delivery if the article's answer is missing
Google indexing status	Use Search Console's URL Inspection tool to check indexing issues or snippet restrictions

Use the provider's current official IP list when configuring access. For blocked requests, record the URL, time, response code, and blocking rule to investigate the cause.

For further crawl and indexing checks, use our technical SEO checklist.

Protect private pages with authentication and permissions

ChatGPT-User and Perplexity-User may retrieve pages in response to a user's question. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them. (OpenAI: ChatGPT-User, Perplexity: Perplexity-User)

Protect internal documents and member content with authentication and viewing permissions. robots.txt asks bots to respect crawl rules, and the file itself is public. For a private page, verify that unauthenticated requests cannot retrieve the content. (IETF: Security considerations)

How can you check crawl settings and page access?

Specify which bots can retrieve public articles and which training crawlers should be blocked. Include excluded paths, then check WAF logs and page responses to verify access.

Please Cite Me! is a reporting tool for SEO agencies that checks page access and content quality and lets you export the results.

If your agency reviews crawl settings and content for client reports, give it a try!

Analyze a page with Please Cite Me! →