How to Block AI Crawlers in WordPress

What Is the Block AI Crawlers feature?

AI companies train their models on web content. To collect this content, they use crawlers (bots) that scan websites and ingest the content. The major AI companies have identified crawlers:

• OpenAI: GPTBot, ChatGPT-User, OAI-SearchBot
• Anthropic: ClaudeBot, Claude-User, Claude-SearchBot
• Google: Google-Extended
• Common Crawl: CCBot
• Meta: Meta-ExternalAgent, Meta-ExternalFetcher
• Perplexity: PerplexityBot
• Apple: Applebot-Extended
• Bytespider: Bytespider (TikTok)
• Amazon: Amazonbot
• Cohere: cohere-ai

The Block AI Crawlers feature blocks all of these from accessing your site.

Why You Need It

AI training has become a contentious issue:

• Consent: Your content may be used to train AI models that compete with you
• Attribution: AI models may use your content without credit
• Server load: AI crawlers are aggressive and can strain your server
• Bandwidth: AI crawlers can consume significant bandwidth
• Terms of service: Some users explicitly want their content excluded from AI training

The Block AI Crawlers feature is for users who want to opt out of AI training.

Trade-offs vs allowing AI crawlers

Blocking AI crawlers has trade-offs:

• Reduced AI search visibility: Google’s AI Overviews and other AI search features may not include your content
• Future AI features: As AI becomes more integrated into search, blocked sites may be disadvantaged
• No clear opt-out from major AI companies: Some companies (notably OpenAI) have stated they respect robots.txt; others have not

For most sites, the trade-off is acceptable. The content you publish is yours to control.

How to Block AI Crawlers in WordPress

1. 1
   Step 1

   Navigate to Settings

   Click into the Classic Monks plugin settings in your WordPress dashboard.

2. 2
   Step 2

   Go to the Security Tab

   Click on the Security menu, then click the Content Protection subtab.

3. 3
   Step 3

   Enable Block AI Crawlers

   Scroll to Block AI Crawlers and toggle on.

   

4. 4
   Step 4

   Save Changes

   Click Save Changes.

5. 5
   Step 5

   Verify

   Visit https://yoursite.com/robots.txt. You should see entries like:

   User-agent: GPTBot
   Disallow: /
   
   User-agent: ClaudeBot
   Disallow: /
   
   User-agent: CCBot
   Disallow: /

6. 6
   Step 6

   Test

   Use a tool like OpenAI’s GPTBot verification or your server logs to verify that the AI crawlers are being blocked.

Configuration Options

No nested options. The blocked crawlers list is maintained by Classic Monks and updated as new crawlers emerge.

What Gets Affected

• The robots.txt: updated to disallow AI crawlers
• The AI crawler requests: blocked at the server level
• The AI search visibility: AI search engines may not include your content
• The server load: reduced (AI crawlers are aggressive)

What Does NOT Get Affected

• The regular search engines: Google, Bing, etc. are not blocked
• The human visitors: not affected
• The WordPress admin: not affected
• The API access: not affected (only the crawlers are blocked)

Advanced Options (Developers)

This feature registers 2 WordPress hooks in ai-bot-blocker.php:

Actions:

• wp_loaded calls CM_AI_Bot_Blocker::block_ai_user_agents() (Blocks known AI bot user agents)

Filters:

• robots_txt calls CM_AI_Bot_Blocker::modify_robots_txt() (Adds AI bot disallow rules to robots.txt (priority 10))

// Hooked in ai-bot-blocker.php
add_action( 'wp_loaded', 'CM_AI_Bot_Blocker::block_ai_user_agents' );

The feature modifies WordPress behavior by registering or removing hooks. Disabling it reverses those changes and WordPress returns to its default behavior.

Troubleshooting

5 problems and how to fix each one.