How do I stop AI bots from scraping my website?

Add a robots.txt file at the root of your domain (yoursite.com/robots.txt) with a User-agent line naming each AI crawler and a Disallow rule under it. This tool builds that block for you. Note that robots.txt is honour-based: major operators like OpenAI, Anthropic and Google obey it, but a bot that ignores rules needs a server-side or firewall block instead.

What is the robots.txt line to block GPTBot?

Two lines: `User-agent: GPTBot` followed by `Disallow: /`. That blocks OpenAI's training crawler across your whole site. To block only one folder, use a path like `Disallow: /private/`. The rule takes effect the next time GPTBot reads your robots.txt.

Does blocking Google-Extended affect my Google Search ranking?

No. Google-Extended is a separate token that only controls whether your content trains Gemini and Vertex AI. Google's own documentation states it does not affect crawling or ranking in Google Search, that is handled by Googlebot, which you would leave alone. You can block Google-Extended and keep full Search visibility.

What is the difference between GPTBot and OAI-SearchBot?

GPTBot collects pages to train OpenAI's models. OAI-SearchBot fetches pages so ChatGPT search can cite them in answers, which can send you referral traffic. ChatGPT-User is a third token for user-triggered fetches. Blocking GPTBot while allowing OAI-SearchBot keeps you out of training data but still eligible for citations.

Can I block AI crawlers without blocking normal search engines?

Yes. AI crawlers use their own user-agent tokens (GPTBot, ClaudeBot, CCBot and so on), separate from Googlebot and Bingbot. Naming only the AI tokens in robots.txt leaves classic search-engine crawling untouched, so your pages still rank in Google and Bing.

Where do I put the generated robots.txt block?

In the plain-text file served at the root of your domain, https://yoursite.com/robots.txt. If you already have a robots.txt, paste these stanzas below your existing rules. On WordPress, an SEO plugin (Yoast, Rank Math) has a robots.txt editor; on static hosts, drop the file in your public/root folder.

Is blocking these crawlers legally enforceable?

robots.txt is a voluntary standard (RFC 9309), not a legal barrier. It records your preference and the named operators publicly commit to honouring it. It does not by itself create a binding contract, and bots that disregard it are not stopped by the file alone. For enforcement you would need server-side blocking and, potentially, terms-of-service or legal measures.

How often is the crawler list updated?

The registry lists 25 crawlers and is re-checked against each operator's own documentation on a roughly quarterly cadence. It was last verified on 2026-07-03. New tokens (operators add and rename bots regularly) are added on review; each row links to the source doc so you can confirm it yourself.

Developer · AI

AI Crawler robots.txt Generator, block GPTBot & other AI bots

Build a ready-to-paste robots.txt block that allows or blocks 25 AI crawlers, GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot and more. Pick per bot or use one-click presets. Runs entirely in your browser, nothing uploaded, sources cited below.

By Induwara Ashinsana— Co-Founder, Ryzera TechnologiesUpdated Jul 3, 2026

Build your AI robots.txt25 crawlers

RFC 9309 · runs in your browser

Presets

Disallow crawlers that harvest your pages for model training; keep AI-search and assistant bots that can send you referral traffic.

Path to control

Use / for the whole site, or a folder like /blog/. Must start with a slash, no spaces.

Crawlers & rules13 of 25 selected

Include	Crawler	Purpose	Your rule
	`GPTBot` OpenAI	Training model training
	`OAI-SearchBot` OpenAI	AI search ChatGPT search citations
	`ChatGPT-User` OpenAI	Assistant user-triggered fetch
	`ClaudeBot` Anthropic	Training model training
	`anthropic-ai` Anthropic	Training legacy training crawler
	`Claude-SearchBot` Anthropic	AI search Claude search citations
	`Claude-User` Anthropic	Assistant user-triggered fetch
	`Google-Extended` Google	Training Gemini and Vertex training
	`Google-CloudVertexBot` Google	Assistant Vertex AI agent fetch
	`PerplexityBot` Perplexity	AI search AI search citations
	`Perplexity-User` Perplexity	Assistant user-triggered fetch
	`CCBot` Common Crawl	Training open training dataset
	`Amazonbot` Amazon	Assistant Alexa answers and indexing
	`Applebot-Extended` Apple	Training Apple Intelligence training
	`Meta-ExternalAgent` Meta	Training model training
	`Meta-ExternalFetcher` Meta	Assistant user-triggered fetch
	`Bytespider` ByteDance	Training model training
	`cohere-ai` Cohere	Training model training
	`MistralAI-User` Mistral	Assistant user-triggered fetch
	`DuckAssistBot` DuckDuckGo	Assistant DuckAssist answers
	`YouBot` You.com	AI search AI search citations
	`Diffbot` Diffbot	Training knowledge-graph training
	`Omgilibot` Webz.io	Training training-data resale
	`ImagesiftBot` ImageSift	Training image dataset training
	`Timpibot` Timpi	Training search and training index

robots.txtValid · 13 rules7 blocked · 6 allowed

# OpenAI, model training (source: platform.openai.com/docs/bots)
User-agent: GPTBot
Disallow: /

# OpenAI, ChatGPT search citations (source: platform.openai.com/docs/bots)
User-agent: OAI-SearchBot
Allow: /

# OpenAI, user-triggered fetch (source: platform.openai.com/docs/bots)
User-agent: ChatGPT-User
Allow: /

# Anthropic, model training (source: support.anthropic.com)
User-agent: ClaudeBot
Disallow: /

# Anthropic, user-triggered fetch (source: support.anthropic.com)
User-agent: Claude-User
Allow: /

# Google, Gemini and Vertex training (source: developers.google.com/search)
User-agent: Google-Extended
Disallow: /

# Perplexity, AI search citations (source: docs.perplexity.ai/guides/bots)
User-agent: PerplexityBot
Allow: /

# Perplexity, user-triggered fetch (source: docs.perplexity.ai/guides/bots)
User-agent: Perplexity-User
Allow: /

# Common Crawl, open training dataset (source: commoncrawl.org/ccbot)
User-agent: CCBot
Disallow: /

# Amazon, Alexa answers and indexing (source: developer.amazon.com/amazonbot)
User-agent: Amazonbot
Allow: /

# Apple, Apple Intelligence training (source: support.apple.com/119829)
User-agent: Applebot-Extended
Disallow: /

# Meta, model training (source: developers.facebook.com/docs/sharing/webmasters)
User-agent: Meta-ExternalAgent
Disallow: /

# ByteDance, model training (source: github.com/ai-robots-txt)
User-agent: Bytespider
Disallow: /

What “Block training, allow AI search” means

Best of both worlds for most publishers: your content still earns citations in ChatGPT search, Perplexity and Google's AI answers, but it is kept out of the raw training corpora.

robots.txt is honour-based, not a firewall

A crawler chooses whether to obey robots.txt. OpenAI, Anthropic, Google, Apple, Amazon and Common Crawl publicly commit to honouring it. Bots that ignore rules need a server-side or WAF block instead, a separate, harder measure.

Generated entirely in your browser, nothing you pick is uploaded.

How it works

This is a template generator, not a numeric calculator. It maps AI-crawler user-agent tokens to Allow or Disallow rules and emits a robots.txt block that follows the Robots Exclusion Protocol standardised as RFC 9309. Every token comes verbatim from the crawler operator's own published documentation, nothing is invented, so the generated file does exactly what each vendor documents.

The registry sorts each crawler into one of three purposes, which drives the presets:

Training (13 bots), harvests pages for model training datasets. Examples: GPTBot, ClaudeBot, Google-Extended, CCBot.
AI search (4 bots), fetches pages so an AI answer engine can cite them, which can send you referral clicks. Examples: OAI-SearchBot, PerplexityBot.
Assistant (8 bots), user-triggered fetches when a person pastes your URL into a chatbot. Examples: ChatGPT-User, Perplexity-User.

The emitter runs four steps:

Preset → rule map."Block all AI" sets every bot to Disallow; "Block training, allow AI search" sets training bots to Disallow and the rest to Allow; "Allow all" emits an explicit Allow line; Custom lets you toggle each token.
Validate the path. The path must begin with / and contain no spaces. An empty selection shows a hint and emits nothing.
Emit one stanza per bot, in a stable operator-then-token order so the output is reproducible: an optional # operator, purpose comment, a User-agent: line, then the chosen Disallow: or Allow: line.
Cross-check. The generated text is re-parsed by an independent validator that confirms every stanza has exactly one User-agent line, exactly one Allow/Disallow line, a known token, and a path starting with /. The green "Valid" badge above only appears when that round-trip passes, the same two-method verification approach used across this site's tools.

One caveat worth stating plainly: robots.txt is honour-based. It records your preference, and the major operators publicly commit to obeying it, but the file is not a firewall. A crawler that disregards robots.txt can only be stopped server-side (a WAF rule, IP block, or user-agent block), a separate, harder measure that is out of scope for this browser-only tool.

Worked examples

Example 1, "Block training, allow AI search" (path /)

Selected: GPTBot, OAI-SearchBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot.
Rule map: training → Disallow, AI-search → Allow.
Result: 4 training bots blocked, 2 search bots allowed.

# OpenAI, model training (source: platform.openai.com/docs/bots)
User-agent: GPTBot
Disallow: /

# OpenAI, ChatGPT search citations (source: platform.openai.com/docs/bots)
User-agent: OAI-SearchBot
Allow: /

# Anthropic, model training (source: support.anthropic.com)
User-agent: ClaudeBot
Disallow: /

# Google, Gemini and Vertex training (source: developers.google.com/search)
User-agent: Google-Extended
Disallow: /

# Perplexity, AI search citations (source: docs.perplexity.ai/guides/bots)
User-agent: PerplexityBot
Allow: /

# Common Crawl, open training dataset (source: commoncrawl.org/ccbot)
User-agent: CCBot
Disallow: /

Example 2, "Block all AI" on /private/, comments off

Selected: GPTBot, ClaudeBot. Path: /private/. Comments: off.
Every selected bot → Disallow: /private/.
Result: two comment-free stanzas scoped to the folder.

User-agent: GPTBot
Disallow: /private/

User-agent: ClaudeBot
Disallow: /private/

Example 3, "Allow all" for the OpenAI family (path /)

Selected: GPTBot, OAI-SearchBot, ChatGPT-User. Preset: Allow all.
Each bot gets an explicit Allow: / line.
Result: an opt-in block that signals you welcome OpenAI's crawlers.

# OpenAI, model training (source: platform.openai.com/docs/bots)
User-agent: GPTBot
Allow: /

# OpenAI, ChatGPT search citations (source: platform.openai.com/docs/bots)
User-agent: OAI-SearchBot
Allow: /

# OpenAI, user-triggered fetch (source: platform.openai.com/docs/bots)
User-agent: ChatGPT-User
Allow: /

Frequently asked questions

Sources & references

Every user-agent token is taken from the operator's own published docs; a handful of less-documented tokens are corroborated against the community ai.robots.txt reference. The registry was last cross-checked on 2026-07-03 and is reviewed on a roughly quarterly cadence. Each row in the tool links directly to its source.

Related tools

AI Bot Blocker (robots.txt)

Generate ready-to-paste robots.txt rules to block AI crawlers, GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot and more, while keeping Google and Bing search untouched. Pick training scrapers vs. AI-search bots, see each bot's SEO impact, copy or download. Runs entirely in your browser.

Open tool

AI JSON Repair

Paste the malformed JSON an LLM returned, Markdown fences, trailing commas, single quotes, Python True/None, comments, or a truncated tail, and get valid, parseable JSON instantly. Lists every fix applied. 100% in your browser, no signup, no API key.

Open tool

AI Carbon Calculator

Estimate the electricity, CO₂e, and water footprint of your ChatGPT, Gemini, or other AI chatbot usage. Enter prompts per day and your country's grid to see energy, carbon, and relatable equivalents, every figure traced to a published source.

Open tool

Comments & feedback

Spotted a bug or want an improvement? Tell us, our team reviews every comment, and good ideas get built. Comments are public and anonymous.

Spotted a new AI crawler, a renamed token, or a bug in the output?

Email me at [email protected] , the registry is updated on review.