How to Block AI Scrapers Without Blocking Google

AI companies send bots to local news sites every day. Some copy (scrape) your articles to train their AI models. Others summarize your stories inside ChatGPT, Perplexity and similar tools, so readers get the news without ever clicking through to your site. That means no pageview, no ad impression and no chance to turn a reader into a subscriber.

You can push back without hurting your Google traffic. The most effective approach uses three layers:

robots.txt asks well-behaved AI bots to stay out.
Cloudflare blocks AI bots before they reach your site, including the ones that ignore robots.txt.
A free registration wall keeps your full stories behind a login, where AI bots can’t get them, while growing your email list.

This guide walks through all three, so you can set them up on any site. If your site runs on WordPress with PaywallProject, AI Shield handles the first two layers for you.

The three kinds of AI bots

Not every AI bot does the same job, and the difference matters:

Training crawlers (like GPTBot and ClaudeBot) collect your content to build future AI models.
AI search crawlers (like OAI-SearchBot and PerplexityBot) build the indexes AI assistants use to answer questions about current events, including your stories.
On-demand fetchers (like ChatGPT-User and Perplexity-User) grab a page the moment someone asks an AI tool about it.

If your concern is losing traffic, block all three. Training bots take your work for free, and the search and fetch bots answer readers’ questions in place of a click. AI tools send very little referral traffic to local news sites today, so blocking them costs you almost nothing.

Layer 1: The recommended robots.txt

Copy the file below. Search engines and every bot not listed are still allowed, so Google, Bing, DuckDuckGo and social media link previews (Facebook, X, LinkedIn, Slack) keep working normally. Replace example.com on the last line with your own domain.

# ===== AI training crawlers: blocked =====
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: CCBot
User-agent: Bytespider
User-agent: Amazonbot
User-agent: cohere-ai
User-agent: cohere-training-data-crawler
User-agent: Diffbot
User-agent: Omgilibot
User-agent: Timpibot
User-agent: ImagesiftBot
User-agent: PanguBot
User-agent: AI2Bot
User-agent: FacebookBot
Disallow: /

# ===== AI search / answer / on-demand fetchers: blocked =====
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: meta-externalfetcher
User-agent: MistralAI-User
User-agent: DuckAssistBot
User-agent: YouBot
Disallow: /

# ===== Search engines and everyone else: allowed =====
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/sitemap_index.xml

A note on Google: The Google-Extended line keeps your content out of Google’s Gemini AI models without affecting your Google Search rankings. Never add Googlebot to the blocked list. That would remove your site from Google Search entirely.

How to add it with Yoast SEO

Log in to your WordPress dashboard.
In the left-hand menu, click Yoast SEO, then click Tools.
Click File editor.
If you see a Create robots.txt file button, click it. If your site already has a robots.txt file, you’ll go straight to the editor.
Note the Sitemap: line Yoast shows. That’s your correct sitemap address. Then delete everything in the robots.txt box, paste in the file above, and put your sitemap address on the last line.
Click Save changes to robots.txt.
Visit yoursite.com/robots.txt in a browser to confirm the new rules are live. If you still see the old version, clear your site’s cache and your Cloudflare cache, then check again.

Don’t see “File editor”? Some hosts turn off file editing inside WordPress for security. Ask your web host to add the file for you, or upload a plain text file named robots.txt to your site’s root folder using FTP or your host’s file manager.

Layer 2: Block AI bots at Cloudflare

robots.txt is a request, not a lock. Some AI bots ignore it. If your site runs through Cloudflare (including the free plan), you can block AI bots before they ever reach your server.

Cloudflare now sorts AI bots into three groups with separate settings: Search, Training and Agent (bots that fetch pages for AI assistants in real time). These replaced the old one-click “Block AI bots” switch in September 2026.

Important warning for ad-supported sites: Cloudflare treats Googlebot, Bingbot and Applebot as “mixed-use” crawlers, because they collect content for both search and AI training. If you set Training to Block or Block on pages with ads, Cloudflare blocks Googlebot too, which can knock your site out of Google Search. Since September 15, 2026, Block on pages with ads has been the default for new domains and for free-plan sites that never changed these settings. Because most news sites show ads, check your settings today even if you’ve never touched them.

Recommended Cloudflare settings for news publishers

Log in to the Cloudflare dashboard and select your domain.
Go to Security → Settings and find the AI bot policies for Search, Training and Agent.
Set Search to Allow. This keeps Google, Bing and other search engines crawling your site.
Set Training to Disallow AI Training. This blocks AI training crawlers at the edge and publishes a “no training” signal to Google, Bing and Apple, while still letting their search crawlers in. Do not choose Block or Block on pages with ads here.
Set Agent to Block. This stops AI assistants from fetching your stories on demand.
Save your changes. Cloudflare will also add its own AI rules to the top of your robots.txt. That’s fine; they work alongside the rules you added in Yoast.
Optional: turn on AI Labyrinth on the same page. It sends AI crawlers that ignore your rules into a maze of decoy pages, wasting their time instead of yours.

Optional: a custom rule for stubborn bots

Cloudflare’s settings may still let some AI search bots through, since they can fall under Search. To hard-block specific bots by name, add a custom firewall rule. The free plan includes a handful of custom rules, and this uses one.

In Cloudflare, go to Security → Security rules (called WAF → Custom rules on some accounts) and click Create rule.
Name it something like “Block AI bots by name.”
Click Edit expression and paste in the expression below.
Under Choose action, select Block, then click Deploy.

(lower(http.user_agent) contains "gptbot") or
(lower(http.user_agent) contains "oai-searchbot") or
(lower(http.user_agent) contains "chatgpt-user") or
(lower(http.user_agent) contains "claudebot") or
(lower(http.user_agent) contains "claude-searchbot") or
(lower(http.user_agent) contains "claude-user") or
(lower(http.user_agent) contains "perplexitybot") or
(lower(http.user_agent) contains "perplexity-user") or
(lower(http.user_agent) contains "meta-externalagent") or
(lower(http.user_agent) contains "meta-externalfetcher") or
(lower(http.user_agent) contains "bytespider") or
(lower(http.user_agent) contains "ccbot") or
(lower(http.user_agent) contains "diffbot") or
(lower(http.user_agent) contains "mistralai-user")

The lower() part matters. Cloudflare’s contains is case-sensitive, so a rule that says “GPTBot” misses a bot that sends “gptbot”. Never add Googlebot, Bingbot, Applebot or DuckDuckBot to this rule.

Check your work

In Google Search Console, run URL Inspection on a recent story and click Test live URL. It should say Google can crawl the page. Check again over the next week for any new crawl errors.
In Cloudflare, open Security → Events (or Analytics) to see which AI bots are being blocked.

One limit to know: AI Overviews in Google Search use Google’s regular search crawler. No robots.txt or Cloudflare setting can opt you out of them without also leaving Google Search, so most publishers accept this trade-off.

The easy way: AI Shield for WordPress

Layers 1 and 2 take some care to set up and to keep current. If your site uses PaywallProject, AI Shield does most of that work for you:

It writes the robots.txt rules. Turn it on and it adds a rule for each AI training crawler and AI search fetcher to the top of your robots.txt. Your sitemap line and your SEO plugin’s rules stay as they are. It works with Yoast SEO and All in One SEO.
It never blocks search engines. Googlebot, Bingbot, Applebot, DuckDuckBot, YandexBot and Baiduspider are always allowed.
It builds the Cloudflare rule. You get a ready-to-copy rule for the same bots, already in the case-insensitive form above. If the list grows past Cloudflare’s 4,096-character limit, AI Shield splits it into several rules.
It tells you when something is wrong. It warns you when a robots.txt file on disk hides its rules, or when another plugin’s Allow rule lets a blocked bot back in.
It stays current. When an update adds a new AI bot, your robots.txt blocks it without any work from you.

You still choose which bots to block, and you still paste the Cloudflare rule into Cloudflare yourself. AI Shield has the same limit as any robots.txt or Cloudflare setup: it can’t catch an AI bot that pretends to be a regular browser. That’s what Layer 3 is for.

robots.txt and Cloudflare stop the bots that identify themselves. The surest way to protect your reporting is to make the full story available only to readers who log in. AI bots don’t sign up for accounts.

You don’t need to charge for it. With PaywallProject, you can start with a free registration wall: readers see the headline and opening paragraphs, then create a free account with their email address to keep reading. Here’s why that works so well for local newsrooms:

It keeps your stories out of AI training. The full article is only delivered to logged-in readers, so AI bots that slip past robots.txt and Cloudflare get the teaser, not the story.
It grows your email list faster than a paywall. Every free account is a new newsletter subscriber. Asking for an email address is a much smaller ask than asking for a credit card.
It brings readers back to your site. Unlike AI tools, your newsletter sends readers straight to your stories, giving you a traffic source you control instead of one that depends on Google or AI platforms.
It makes you more valuable to advertisers. A large, local, logged-in audience and a growing newsletter list are exactly what local businesses want to reach.
It keeps you in Google. Your site tells Google which content requires a login, so your stories still appear in search results without being treated as hidden content.

A free registration wall is also the best first step toward paid subscriptions. Once your newsletter list is large and your brand is well-known, you can introduce a paid tier. Even then, we recommend keeping some stories available with free registration so your list keeps growing. And for nonprofit newsrooms, a registration wall pairs naturally with membership and donation asks instead of a paywall.

Want to see how a free registration wall would work on your site? Book a 30-minute call with PaywallProject.

Keep your defenses current

New AI bots launch and rename themselves regularly, so revisit your robots.txt and Cloudflare rules every few months. The community-maintained ai.robots.txt project on GitHub tracks more than 150 AI bots and is a good place to check for new names to add. If you use AI Shield, PaywallProject updates add new bots for you. Check the Cloudflare rule on the AI Shield tab after each update, since Cloudflare doesn’t update it on its own.

Want to keep your reporting out of AI training without the manual work? See how AI Shield works, then compare plans or get started with PaywallProject.