Should you block AI crawlers?
Training and search are two different doors. You can close one and keep the other open.
Facts checked 3 Oct 2026 · by the AdSworn team
It depends what you're blocking. AI companies run separate crawlers for model training and for search answers. Blocking a training crawler (OpenAI's GPTBot, Anthropic's ClaudeBot, Google-Extended) keeps your content out of future training without, by the companies' own accounts, removing you from their search: Google says Google-Extended doesn't affect Search inclusion or ranking. Blocking a search crawler (OAI-SearchBot, Claude-SearchBot) can remove you from that assistant's answers and the traffic they send.
Who's who
| Crawler | Company | Used for | If you block it |
|---|---|---|---|
| GPTBot | OpenAI | Training models | Content signalled as not for training; doesn't affect ChatGPT search |
| OAI-SearchBot | OpenAI | Surfacing sites in ChatGPT search | You won't appear in ChatGPT search answers |
| ChatGPT-User | OpenAI | Actions a person asks ChatGPT to take | Those user-requested visits stop |
| ClaudeBot | Anthropic | Training models | Excluded from future training data |
| Claude-SearchBot | Anthropic | Improving Claude's search results | May reduce visibility and accuracy in Claude's search answers |
| Claude-User | Anthropic | Fetching pages when a person asks Claude | May reduce visibility for user-directed search |
| Google-Extended | Whether content may train future Gemini models | No effect on Google Search inclusion or ranking |
How to block one
Each company says its crawlers respect robots.txt. To block a single bot, add its name and a Disallow line, for example User-agent: GPTBot then Disallow: /. Leave the search crawlers you want to appear in untouched. Google's AI Overviews and AI Mode use normal Googlebot crawling, so blocking Googlebot removes you from Google Search entirely.
How to decide
| If you want… | Then |
|---|---|
| Traffic and citations from AI assistants | Allow the search crawlers (OAI-SearchBot, Claude-SearchBot, Claude-User, Googlebot) |
| Your content kept out of model training | Block the training crawlers (GPTBot, ClaudeBot, Google-Extended) |
| Both | Block training, allow search |
| Neither | Block them all, and accept you won't appear in those assistants' answers |
What AdSworn does, and why
AdSworn allows every crawler on its public pages: the point of a listing is to be found, by people and by the assistants they ask. For publishers, a page's value to a sponsor comes from its real readers, which AdSworn reads from Google Search Console and Google Analytics, so a crawler choice doesn't change a page's price.
- Publisher connects GoogleRead-only sign-in to Search Console (and Analytics if they use it). No passwords shared.
- AdSworn reads the numbersClicks from Google, the searches behind each page and page views. For this page: 21 clicks from Google in the last 4 weeks.
- Buyers see the proofEvery figure shows its source and the day it was read (3 Oct 2026 for this site).
- Banner ad at $75 / 4 weeksyou get $69
- Featured post at $1,300 / yearyou get $1,196
Starter rates from what sites charge for ads they sell directly: $10–20 per 1,000 views (Google). Banner $15 per 1,000 page views ($25 minimum, 4 weeks); featured post $20 per 1,000 over a year ($150 minimum). If both sell all year. You set your own; we take 8% only when you get paid.
Straight answers
Does blocking GPTBot remove my site from ChatGPT?
OpenAI says blocking GPTBot signals your content shouldn't be used for training; appearing in ChatGPT search depends on OAI-SearchBot, a separate crawler.
Does blocking Google-Extended hurt my Google rankings?
No. Google says Google-Extended doesn't affect a site's inclusion in Google Search and isn't a ranking signal.
How do I block ClaudeBot?
Add 'User-agent: ClaudeBot' and 'Disallow: /' to your robots.txt. Anthropic says its crawlers respect robots.txt.