How do I stop AI bots from scraping my website?

Add a robots.txt file at the root of your domain (yoursite.com/robots.txt) with a User-agent line naming each AI crawler and a Disallow rule under it. This tool builds that block for you. Note that robots.txt is honour-based: major operators like OpenAI, Anthropic and Google obey it, but a bot that ignores rules needs a server-side or firewall block instead.

What is the robots.txt line to block GPTBot?

Two lines: `User-agent: GPTBot` followed by `Disallow: /`. That blocks OpenAI's training crawler across your whole site. To block only one folder, use a path like `Disallow: /private/`. The rule takes effect the next time GPTBot reads your robots.txt.

Does blocking Google-Extended affect my Google Search ranking?

No. Google-Extended is a separate token that only controls whether your content trains Gemini and Vertex AI. Google's own documentation states it does not affect crawling or ranking in Google Search — that is handled by Googlebot, which you would leave alone. You can block Google-Extended and keep full Search visibility.

What is the difference between GPTBot and OAI-SearchBot?

GPTBot collects pages to train OpenAI's models. OAI-SearchBot fetches pages so ChatGPT search can cite them in answers, which can send you referral traffic. ChatGPT-User is a third token for user-triggered fetches. Blocking GPTBot while allowing OAI-SearchBot keeps you out of training data but still eligible for citations.

Can I block AI crawlers without blocking normal search engines?

Yes. AI crawlers use their own user-agent tokens (GPTBot, ClaudeBot, CCBot and so on), separate from Googlebot and Bingbot. Naming only the AI tokens in robots.txt leaves classic search-engine crawling untouched, so your pages still rank in Google and Bing.

Where do I put the generated robots.txt block?

In the plain-text file served at the root of your domain — https://yoursite.com/robots.txt. If you already have a robots.txt, paste these stanzas below your existing rules. On WordPress, an SEO plugin (Yoast, Rank Math) has a robots.txt editor; on static hosts, drop the file in your public/root folder.

Is blocking these crawlers legally enforceable?

robots.txt is a voluntary standard (RFC 9309), not a legal barrier. It records your preference and the named operators publicly commit to honouring it. It does not by itself create a binding contract, and bots that disregard it are not stopped by the file alone. For enforcement you would need server-side blocking and, potentially, terms-of-service or legal measures.

How often is the crawler list updated?

The registry lists 25 crawlers and is re-checked against each operator's own documentation on a roughly quarterly cadence. It was last verified on 2026-07-03. New tokens (operators add and rename bots regularly) are added on review; each row links to the source doc so you can confirm it yourself.

Developer · AI

AI Crawler robots.txt Generator — block GPTBot & other AI bots

Build a ready-to-paste robots.txt block that allows or blocks 25 AI crawlers — GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot and more. Pick per bot or use one-click presets. Runs entirely in your browser, nothing uploaded, sources cited below.

By Induwara Ashinsana— Executive Director, Ryzera TechnologiesUpdated Jul 3, 2026

Build your AI robots.txt25 crawlers

RFC 9309 · runs in your browser

Presets

Disallow crawlers that harvest your pages for model training; keep AI-search and assistant bots that can send you referral traffic.

Path to control

Use / for the whole site, or a folder like /blog/. Must start with a slash, no spaces.

Crawlers & rules13 of 25 selected

Include	Crawler	Purpose	Your rule
	`GPTBot` OpenAI	Training model training
	`OAI-SearchBot` OpenAI	AI search ChatGPT search citations
	`ChatGPT-User` OpenAI	Assistant user-triggered fetch
	`ClaudeBot` Anthropic	Training model training
	`anthropic-ai` Anthropic	Training legacy training crawler
	`Claude-SearchBot` Anthropic	AI search Claude search citations
	`Claude-User` Anthropic	Assistant user-triggered fetch
	`Google-Extended` Google	Training Gemini and Vertex training
	`Google-CloudVertexBot` Google	Assistant Vertex AI agent fetch
	`PerplexityBot` Perplexity	AI search AI search citations
	`Perplexity-User` Perplexity	Assistant user-triggered fetch
	`CCBot` Common Crawl	Training open training dataset
	`Amazonbot` Amazon	Assistant Alexa answers and indexing
	`Applebot-Extended` Apple	Training Apple Intelligence training
	`Meta-ExternalAgent` Meta	Training model training
	`Meta-ExternalFetcher` Meta	Assistant user-triggered fetch
	`Bytespider` ByteDance	Training model training
	`cohere-ai` Cohere	Training model training
	`MistralAI-User` Mistral	Assistant user-triggered fetch
	`DuckAssistBot` DuckDuckGo	Assistant DuckAssist answers
	`YouBot` You.com	AI search AI search citations
	`Diffbot` Diffbot	Training knowledge-graph training
	`Omgilibot` Webz.io	Training training-data resale
	`ImagesiftBot` ImageSift	Training image dataset training
	`Timpibot` Timpi	Training search and training index

robots.txtValid · 13 rules7 blocked · 6 allowed

# OpenAI — model training (source: platform.openai.com/docs/bots)
User-agent: GPTBot
Disallow: /

# OpenAI — ChatGPT search citations (source: platform.openai.com/docs/bots)
User-agent: OAI-SearchBot
Allow: /

# OpenAI — user-triggered fetch (source: platform.openai.com/docs/bots)
User-agent: ChatGPT-User
Allow: /

# Anthropic — model training (source: support.anthropic.com)
User-agent: ClaudeBot
Disallow: /

# Anthropic — user-triggered fetch (source: support.anthropic.com)
User-agent: Claude-User
Allow: /

# Google — Gemini and Vertex training (source: developers.google.com/search)
User-agent: Google-Extended
Disallow: /

# Perplexity — AI search citations (source: docs.perplexity.ai/guides/bots)
User-agent: PerplexityBot
Allow: /

# Perplexity — user-triggered fetch (source: docs.perplexity.ai/guides/bots)
User-agent: Perplexity-User
Allow: /

# Common Crawl — open training dataset (source: commoncrawl.org/ccbot)
User-agent: CCBot
Disallow: /

# Amazon — Alexa answers and indexing (source: developer.amazon.com/amazonbot)
User-agent: Amazonbot
Allow: /

# Apple — Apple Intelligence training (source: support.apple.com/119829)
User-agent: Applebot-Extended
Disallow: /

# Meta — model training (source: developers.facebook.com/docs/sharing/webmasters)
User-agent: Meta-ExternalAgent
Disallow: /

# ByteDance — model training (source: github.com/ai-robots-txt)
User-agent: Bytespider
Disallow: /

What “Block training, allow AI search” means

Best of both worlds for most publishers: your content still earns citations in ChatGPT search, Perplexity and Google's AI answers, but it is kept out of the raw training corpora.

robots.txt is honour-based, not a firewall

A crawler chooses whether to obey robots.txt. OpenAI, Anthropic, Google, Apple, Amazon and Common Crawl publicly commit to honouring it. Bots that ignore rules need a server-side or WAF block instead — a separate, harder measure.

Generated entirely in your browser — nothing you pick is uploaded.

How it works

This is a template generator, not a numeric calculator. It maps AI-crawler user-agent tokens to Allow or Disallow rules and emits a robots.txt block that follows the Robots Exclusion Protocol standardised as RFC 9309. Every token comes verbatim from the crawler operator's own published documentation — nothing is invented — so the generated file does exactly what each vendor documents.

The registry sorts each crawler into one of three purposes, which drives the presets:

Training (13 bots) — harvests pages for model training datasets. Examples: GPTBot, ClaudeBot, Google-Extended, CCBot.
AI search (4 bots) — fetches pages so an AI answer engine can cite them, which can send you referral clicks. Examples: OAI-SearchBot, PerplexityBot.
Assistant (8 bots) — user-triggered fetches when a person pastes your URL into a chatbot. Examples: ChatGPT-User, Perplexity-User.

The emitter runs four steps:

Preset → rule map."Block all AI" sets every bot to Disallow; "Block training, allow AI search" sets training bots to Disallow and the rest to Allow; "Allow all" emits an explicit Allow line; Custom lets you toggle each token.
Validate the path. The path must begin with / and contain no spaces. An empty selection shows a hint and emits nothing.
Emit one stanza per bot, in a stable operator-then-token order so the output is reproducible: an optional # operator — purpose comment, a User-agent: line, then the chosen Disallow: or Allow: line.
Cross-check. The generated text is re-parsed by an independent validator that confirms every stanza has exactly one User-agent line, exactly one Allow/Disallow line, a known token, and a path starting with /. The green "Valid" badge above only appears when that round-trip passes — the same two-method verification approach used across this site's tools.

One caveat worth stating plainly: robots.txt is honour-based. It records your preference, and the major operators publicly commit to obeying it, but the file is not a firewall. A crawler that disregards robots.txt can only be stopped server-side (a WAF rule, IP block, or user-agent block) — a separate, harder measure that is out of scope for this browser-only tool.

Worked examples

Example 1 — "Block training, allow AI search" (path /)

Selected: GPTBot, OAI-SearchBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot.
Rule map: training → Disallow, AI-search → Allow.
Result: 4 training bots blocked, 2 search bots allowed.

# OpenAI — model training (source: platform.openai.com/docs/bots)
User-agent: GPTBot
Disallow: /

# OpenAI — ChatGPT search citations (source: platform.openai.com/docs/bots)
User-agent: OAI-SearchBot
Allow: /

# Anthropic — model training (source: support.anthropic.com)
User-agent: ClaudeBot
Disallow: /

# Google — Gemini and Vertex training (source: developers.google.com/search)
User-agent: Google-Extended
Disallow: /

# Perplexity — AI search citations (source: docs.perplexity.ai/guides/bots)
User-agent: PerplexityBot
Allow: /

# Common Crawl — open training dataset (source: commoncrawl.org/ccbot)
User-agent: CCBot
Disallow: /

Example 2 — "Block all AI" on /private/, comments off

Selected: GPTBot, ClaudeBot. Path: /private/. Comments: off.
Every selected bot → Disallow: /private/.
Result: two comment-free stanzas scoped to the folder.

User-agent: GPTBot
Disallow: /private/

User-agent: ClaudeBot
Disallow: /private/

Example 3 — "Allow all" for the OpenAI family (path /)

Selected: GPTBot, OAI-SearchBot, ChatGPT-User. Preset: Allow all.
Each bot gets an explicit Allow: / line.
Result: an opt-in block that signals you welcome OpenAI's crawlers.

# OpenAI — model training (source: platform.openai.com/docs/bots)
User-agent: GPTBot
Allow: /

# OpenAI — ChatGPT search citations (source: platform.openai.com/docs/bots)
User-agent: OAI-SearchBot
Allow: /

# OpenAI — user-triggered fetch (source: platform.openai.com/docs/bots)
User-agent: ChatGPT-User
Allow: /

Frequently asked questions

Sources & references

Every user-agent token is taken from the operator's own published docs; a handful of less-documented tokens are corroborated against the community ai.robots.txt reference. The registry was last cross-checked on 2026-07-03 and is reviewed on a roughly quarterly cadence. Each row in the tool links directly to its source.

Related tools

LiveAI

AI Bot Blocker (robots.txt)

Generate ready-to-paste robots.txt rules to block AI crawlers — GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot and more — while keeping Google and Bing search untouched. Pick training scrapers vs. AI-search bots, see each bot's SEO impact, copy or download. Runs entirely in your browser.

Open tool

LiveAI

Perplexity Calculator

Compute language-model perplexity from token probabilities, cross-entropy loss, or log-likelihood, with nats and bits-per-token conversions. Step-by-step, matches PyTorch, runs entirely in the browser.

Open tool

LiveAI

System Prompt Generator

Turn a short form into a structured system prompt — role, task, tone, output format, guardrails and few-shot examples — for ChatGPT Custom Instructions, a Custom GPT, or the Claude, OpenAI or Gemini system field. Generic Markdown or Claude XML style, with a live preview and copy/download. Runs entirely in your browser; nothing is uploaded, no AI rewriting.

Open tool

Rate this tool

Be the first to rate

Comments & feedback

Spotted a bug or want an improvement? Tell us — our team reviews every comment, and good ideas get built. Comments are public and anonymous.

Spotted a new AI crawler, a renamed token, or a bug in the output?

Email me at [email protected] — the registry is updated on review.