How to Block AI Crawlers on WordPress (and Which to Allow)

By Rishabh Jha & Anuj Kamble, WisdmLabs · Updated 27 October 2026 · 7 min read

AI crawlers are the bots that fetch web pages for AI companies. They do three jobs: collecting training data, building an AI search index, and fetching a page live when a user asks. Blocking a training crawler keeps you in AI search; blocking a search crawler takes you out of it. Decide per bot, in robots.txt.

“Should I block GPTBot?” splits SEO forums down the middle, and both sides are often arguing about the wrong bot. Before you block AI crawlers on WordPress, one rule: never add a blanket block. A wrong line in robots.txt can hide your whole site from Google. Back up your current file first.

Which AI crawlers visit WordPress sites?

Each AI company runs several bots, and each bot has one job. This AI crawlers list is taken from each company’s own documentation, checked 16 October 2026.

Most of these bots follow robots.txt. The exceptions are the live fetchers: ChatGPT-User, Perplexity-User and Meta’s Meta-ExternalFetcher may ignore it, because a person asked for the page. robots.txt also only affects future crawling: content already collected isn’t removed.

What do AI crawlers actually take from your site?

Mostly training data. In July 2025, nearly 80% of AI bot crawling on Cloudflare’s network was for training, with user-requested fetches under 5% (Cloudflare).

Very little traffic comes back. In the first week of August 2025, Cloudflare counted roughly 50,000 Anthropic crawls for every visit Claude referred, 887 for OpenAI and 118 for Perplexity.

These figures move month to month, so treat them as a snapshot, not a constant. Training bots take a lot and send little back; search bots are the ones that can send you visitors.

Should you block AI crawlers on WordPress? A decision table

Start from what you want, not from a list of bots to block:

Blocking has a cost to weigh. A Rutgers and Wharton working paper (April 2026) found that large news publishers who blocked generative AI bots saw less website traffic than those who didn’t. It isn’t peer-reviewed, and it studied big publishers, not small sites. To see whether AI search names you at all, measure your Share of Model.

Blocking Googlebot is never the answer. AI Overviews draw on Google’s main search index, and Google-Extended doesn’t control them. To limit what AI Overviews show from a page, use the nosnippet, data-nosnippet or max-snippet controls (Google).

How to edit robots.txt on WordPress safely

Back up first: open https://yoursite.com/robots.txt and save a copy before you change anything. If a change goes wrong, you can put the old version straight back.

WordPress serves a virtual robots.txt unless a real file exists in your site root, and a real file overrides any plugin that edits the virtual one. This GPTBot robots.txt example opts out of training but stays in AI search, and keeps WordPress’s default rules:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: CCBot
Disallow: /

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://yoursite.com/sitemap.xml

A bot named in its own group ignores the * group, so repeat any rules you still want it to follow. Use your real sitemap URL.

Watch for blocks you didn’t mean to add. The author of a 1,500-site crawl shared on Hacker News found that many AI bots were blocked by accident, through old blanket User-agent: * rules or security plugins. Never put Disallow: / under User-agent: *: that blocks Googlebot too.

1. After saving, open yoursite.com/robots.txt and check the rules appear exactly as written.
2. In Google Search Console, open the robots.txt report. It shows when Google last fetched the file and any parsing issues, and you can request a recrawl.
3. Allow about 24 hours: OpenAI and Perplexity both say their systems can take that long to pick up changes.

Ranklance’s AI Crawler dashboard (paid plans) shows these rules in one screen and merges them into your robots.txt.

Using Cloudflare? Check its AI crawler settings

Many WordPress sites sit behind Cloudflare, which can block AI bots before they ever see your robots.txt. In July 2026, Cloudflare split its AI controls into three groups: Search, Agent and Training.

From 15 September 2026, new domains added to Cloudflare block Training and Agent crawlers by default on pages that show ads. Search crawlers stay allowed. The defaults were discussed widely, including on Hacker News.

Watch out: Cloudflare says that if you block Training, crawlers that do both search and training, such as Googlebot, Applebot and Bingbot, are blocked too. Check your Cloudflare AI crawler settings under Security before you rely on them, and don’t block Training there if you need Google Search.

Your robots.txt and your Cloudflare settings both decide who gets in, so check both.

What if a bot ignores robots.txt or overloads your server?

robots.txt is a request, not a lock. Named bots from the big AI companies follow it, but plenty of scrapers don’t: one developer on Hacker News found that most of the crawlers hitting their server ignored it completely.

Load is a real issue too. A site owner reported that ClaudeBot sent almost a million requests in one day. Anthropic says its bots support the non-standard Crawl-delay rule, so you can slow them down rather than block them.

• Slow, don’t block: add Crawl-delay: 10 under a bot’s user agent, if that bot supports it.
• Bad actors: use your host’s firewall or a security plugin to block abusive IP addresses.
• Check before blocking by IP: OpenAI and Common Crawl publish their crawlers’ IP ranges, so you can tell real bots from fakes. Anthropic warns that IP blocks can stop its bots reading your robots.txt.

Use robots.txt for bots that listen, and your firewall for the ones that don’t.

Check which AI crawlers visit your site

Your server access logs record each crawler’s user agent. Search them for the names in the table to see which bots visit and how often. Google-Extended and Applebot-Extended never appear, because they’re robots.txt tokens, not bots. User agents can be faked, so treat counts as estimates.

Crawler visits don’t show up in Google Analytics, which counts people. To see the human visits AI tools send you, track AI traffic in GA4. Logs show who reads your site; GA4 shows who visits because of it.

AI crawler checklist

1. Back up your current robots.txt.
2. Decide your goal: AI search, no training, or neither.
3. Name each bot in its own group; never Disallow: / under User-agent: *.
4. Check the live file and Search Console’s robots.txt report.
5. Check your Cloudflare AI crawler settings, if you use Cloudflare.
6. Review your logs monthly for new or misbehaving bots.

FAQ