Are you blocking AI shopping agents? A complete guide to robots.txt, CDN and WAF

Direct answer: A store can block AI shopping agents without knowing it, and the block usually sits in a CDN or firewall rule rather than in robots.txt. The vendors publish separate user agents for training, search and live user requests. That means you can refuse training crawlers (GPTBot, ClaudeBot) and still let search and shopping agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-SearchBot, Claude-User) reach your product pages. Verify agents by IP range, reverse DNS or a Web Bot Auth signature, never by user-agent string alone.

This guide lists the agents that matter to a retailer, with their user agents taken from vendor documentation. It also gives a decision table, a sample robots.txt, the settings that matter in Cloudflare, Akamai, Fastly and AWS WAF, and a test you can run in ten minutes.

Last verified: 24 October 2026. Vendors change user agents and defaults often, so every fact below names its source.

Why this matters more than it looks

An AI assistant that cannot fetch your product page cannot quote your price, confirm your stock or send a shopper to your checkout. It will recommend a store it can read. Nothing shows up in your analytics either, because a blocked agent never loads your tags. Google's own guidance for AI Overviews and AI Mode lists "ensuring that crawling is allowed in robots.txt, and by any CDN or hosting infrastructure" among the basics (Google Search Central).

Our Agent Commerce Index (Q4 2026) read 225 UK and US retailers as a real browser and as GPTBot, ChatGPT-User, PerplexityBot and ClaudeBot. Of the 116 stores where the basket test could run, only 8 let a page-fetching agent reach checkout, against 96 for an agent browsing like a person. A further 15 retailers could not be scored at all, and the Index notes that "most refuse automated readers outright". The Index explainer covers how that was measured. Access is the first of the five dimensions in our scoring specification (checks ACC-01 to ACC-14) for this reason.

There are four layers where an agent gets stopped, and you need to check all of them:

1. robots.txt: a request to well-behaved crawlers.
2. CDN or WAF bot management: blocks, challenges or rate limits applied at the edge.
3. JavaScript challenges and captchas: a small delay for a person, usually a dead end for an agent.
4. Interstitials: cookie walls, age gates and geo-redirects that hide product content.

The agents that matter to a retailer

Each entry below comes from the vendor's own documentation. Version numbers in user-agent strings change, so match on the product token (for example OAI-SearchBot), not the full string.

OpenAI

Source: Overview of OpenAI Crawlers. OpenAI says each setting "is independent of the others".

ChatGPT Work's Cloud browser, which drives a real browser for tasks, signs its requests with Web Bot Auth. OpenAI's allowlisting guide gives the Signature-Agent value "https://chatgpt.com", Cloudflare's identifiers for it (bot tag chatgpt-agent, detection ID 129220581) and the matching entries in Akamai, HUMAN and Vercel. OpenAI also says that at launch the Cloud browser "cannot sign in to websites or complete payments".

Perplexity

Source: Perplexity Crawlers.

Anthropic (Claude)

Source: Claude Help Center, updated 7 April 2026. Anthropic says its bots honour robots.txt and the non-standard Crawl-delay, and that blocking IP addresses instead "may not work correctly or persistently guarantee an opt-out". The page links an IP list but does not publish full user-agent strings, so match on the token.

Google

Source: Google's common crawlers.

Product data for Google's shopping surfaces also comes from Merchant Center, not only from crawling.

Microsoft (Bing)

Source: Bing Webmaster Tools: overview of Bing crawlers.

Bing says to verify bingbot by a reverse DNS lookup ending in search.msn.com, then a forward lookup back to the same IP (Verifying Bingbot).

Apple

Source: About Applebot, published 4 September 2026.

Training or shopping: the decision table

The commercial question for a retailer is simple. Does this agent help a shopper find and buy from you, or does it only take content? The table gives our recommended default for a typical retailer. Brands with valuable original content may reasonably choose differently on the training rows.

A common mistake is the opposite pattern: allowing GPTBot by accident (no rule) while a WAF rule blocks ChatGPT-User. That leaves you contributing training data while being invisible to shoppers.

A note on our own scoring: the free score also reads robots.txt for GPTBot, ClaudeBot and Google-Extended (checks ACC-02, ACC-07 and ACC-09). Each carries a small weight (0.5 of the 22 weight points in the Access dimension), so a deliberate training opt-out costs a little there. The firewall checks weigh far more: ACC-10 (product pages return 200 to agent identities) carries 8 and caps the whole score at 40 when it fails.

A sample robots.txt for a retailer

This example allows search and shopping agents, opts out of the training tokens, and keeps the usual retail exclusions. Remove the training block if you want to allow training. robots.txt is a request, not an access control, and some user-triggered agents do not follow it.

# Search engines and AI search
User-agent: Googlebot
User-agent: bingbot
User-agent: Applebot
User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: Claude-SearchBot
Allow: /
Disallow: /checkout/
Disallow: /account/
Disallow: /cart/
Disallow: /*?*sort=
Disallow: /*?*filter=

# User-triggered assistants: product, category and policy pages
User-agent: ChatGPT-User
User-agent: Perplexity-User
User-agent: Claude-User
Allow: /
Disallow: /account/

# Training opt-outs (a business choice)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

# Everyone else
User-agent: *
Disallow: /checkout/
Disallow: /account/

Sitemap: https://www.example.com/sitemap.xml

Notes on the choices:

• Grouping several User-agent lines over one rule set is valid under the robots.txt standard (RFC 9309). A crawler follows only the group (or groups) that name it, so a named bot ignores your * rules entirely. Repeat any exclusions you need in each group.
• We leave basket and checkout open to user-triggered agents in this example, because an agent buying for a shopper needs them. Disallowing /cart/ for search crawlers is normal housekeeping.
• Shopify stores edit robots.txt through the robots.txt.liquid template. Shopify's help page says bot management at the network layer "is handled directly for stores on Shopify", and that product data for agentic storefronts such as ChatGPT or Microsoft Copilot is syndicated through Shopify Catalog "independently of /robots.txt".
• EU stores: EU copyright rules let rightholders reserve their content from text and data mining "in an appropriate manner, such as machine-readable means" (Directive (EU) 2019/790, Article 4). Many businesses use training-token opt-outs as part of that reservation. Take legal advice on whether that is enough for you; this guide is not legal advice.

CDN and WAF settings

robots.txt is rarely where agents actually get blocked. Edge security is. Check each of the following.

Cloudflare

Source: Block AI Bots and Verified bots (last updated 8 October 2026).

• AI bot policies. Cloudflare groups AI traffic as Search ("crawlers that collect or index your content to answer questions about it later"), Agent ("automated activity acting in real time on a person's behalf, such as chat fetch bots and browser-use agents") and Training. Each can be set to Block on all pages, Block on pages with ads, or Allow, under Security Settings > Configure AI bot policies.
• New defaults. Cloudflare's documentation says that from 15 September 2026, new domains block Training and Agent bots on pages that display ads, while Search remains allowed. The legacy "Block AI bots" setting is being deprecated. Mixed-purpose crawlers that combine Search and Training are blocked by any configuration that blocks training.
• For a retailer: set Search: Allow and Agent: Allow, and make Training a business choice. Then check that no custom WAF rule, rate limit or bot setting overrides this on product pages.
• Signed agents are verified bots. As of 1 July 2026, Cloudflare treats signed agents as verified bots, with a "Direct versus Intermediary" field showing who can operate them. OpenAI documents a WAF custom rule that allows ChatGPT Work's Cloud browser by Bot Detection ID 129220581.
• Protect checkout without blanket blocks. Use managed challenges for high-risk requests on payment endpoints and exempt verified bots, rather than challenging all automation.

Cloudflare also runs a separate public scanner that grades agent-facing standards; our guide to Cloudflare's Agent Readiness score explains why it does not test whether agents get past your firewall.

Akamai

Source: Akamai: Redefine trust with Web Bot Authentication (10 November 2025).

• Akamai says Web Bot Auth "is already supported by Akamai's bot & abuse solutions", including App & API Protector. The edge validates the signature under RFC 9421 and classifies valid requests as validated-bot traffic. You then choose allow, challenge, delay or block for that traffic.
• OpenAI's allowlisting guide says Akamai lists ChatGPT Work's Cloud browser as the bot "ChatGPT Agent" in its Artificial Intelligence bots category, which you can allow through a custom bot category.
• Review the actions on AI categories in your own bot policy rather than assuming a default.
• Akamai has also announced support for Visa's Trusted Agent Protocol (17 December 2025), which helps merchants tell "whether an agent is browsing or paying".

Fastly

Source: Managing bot traffic and Next-Gen WAF system signals.

• In Bot Management, actions are set per bot category and per bot identity under Security > Bot Management > Protection. They are enforced only when Protection mode is set to Block.
• Fastly exposes VCL variables such as fastly.bot.category.is_ai_crawler, fastly.bot.category.is_ai_fetcher and fastly.bot.category.is_verified, and Next-Gen WAF signals including VERIFIED-BOT.AI-FETCHER and SUSPECTED-BOT.AI-FETCHER.
• For a retailer, let verified AI fetchers reach product pages and apply your stricter rules to suspected ones. Fastly's own documentation example returns a 402 "AI Licensing Required" to AI crawlers and fetchers. Do not copy that example onto a store without thinking: it would also block shopping fetchers.
• We did not find Web Bot Auth verification in Fastly's public documentation as of 24 October 2026. Ask Fastly if you need it.

AWS WAF

Source: AWS WAF Bot Control rule group and AWS WAF Bot Control.

• Bot Control's CategoryAI rule labels AI bots with awswaf:managed:aws:bot-control:bot:category:ai. Unlike the other category rules, it "applies the action to all matches, regardless of whether the bots are verified or unverified", and its action is Block. An untouched default can therefore block legitimate AI search and fetch bots.
• AWS added Web Bot Auth support on 21 November 2025. When a request carries the web_bot_auth:verified label, the category rules do not match, so verified signed agents get through. At launch this was CloudFront only; the current documentation says it covers CloudFront distributions and Regional resources in the commercial AWS Regions, and needs rule group Version_4.0 or later.
• For a retailer: override CategoryAI to Count, then write explicit allow rules for the AI search and user agents you want, matched by label or by the vendor's IP set. Perplexity's documentation describes the IP-set-plus-user-agent pattern for AWS WAF.

Other layers that block agents

• Hosted platforms run their own bot protection in front of your store. Shopify says this is handled for you and does not recommend a proxy in front of Shopify. You usually cannot change it, but you can test it.
• Captcha and "are you human" interstitials on product or basket pages (spec checks ACC-12 and TRX-03).
• Cookie walls and age gates that hide price or product content until a click (ACC-14). These are common on UK and EU sites because of consent requirements, and in age-restricted categories everywhere. Serve product content in the HTML behind a consent banner that overlays the page rather than replacing it.
• Geo-redirects. AI fetchers run from their vendors' data centres, which may not be in your shopper's country (check the published IP lists). If you geo-redirect by IP, an agent asking about your UK prices may be sent to your US site or a country picker.

Web Bot Auth in brief

User-agent strings can be spoofed. IP lists help but change, and they are only as good as your process for refreshing them. Web Bot Auth lets an agent sign each HTTP request with a private key under the HTTP Message Signatures standard (RFC 9421), publishing its public keys in a directory at /.well-known/http-message-signatures-directory.

What a signed request carries, per OpenAI's documentation:

• a Signature header,
• a Signature-Input header, which Cloudflare's implementation requires to include tag="web-bot-auth", a keyid, and created and expires timestamps,
• a Signature-Agent header pointing at the key directory, for ChatGPT "https://chatgpt.com".

In practice you rarely verify these yourself. Cloudflare, Akamai and AWS WAF verify signatures at the edge and expose the result as a verified-bot classification or label. Your job is to make sure your rules use that classification and that intermediate proxies do not strip the three headers, which is the first troubleshooting step OpenAI lists.

How to test your store in ten minutes

1. Run the free score. The Agent Conversion Score reads your robots.txt and fetches your homepage and up to five product pages as a real browser and as GPTBot, ChatGPT-User, PerplexityBot and ClaudeBot (checks ACC-01 to ACC-14). For a quicker first look, the AI access checker compares what your robots.txt says with what your server actually returns.
2. Fetch a product page as each agent. Compare status codes and page size with a browser request:

URL="https://www.example.com/products/example"
for UA in \
  "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot" \
  "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" \
  "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)" \
  "Claude-User" \
  "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0 Safari/537.36"
do
  curl -s -o /tmp/page.html -w "%{http_code} %{size_download}B  ${UA:0:60}\n" -A "$UA" "$URL"
  grep -c '"@type": *"Product"' /tmp/page.html
done

Anthropic does not publish a full Claude-User string, so the bare token is a stand-in.

3. Read the results carefully. A 403, a 429 or a much smaller page than the browser version suggests a block or challenge. A pass is less conclusive: some WAFs check IP ranges, so a spoofed agent string from your laptop may be treated differently from the real agent, in either direction. Treat this as a first screen.
4. Check your CDN's security events, filtered by the agent tokens above. This is the definitive view of what real agent traffic received.
5. Ask the assistant. In ChatGPT, Perplexity or Claude, paste a product URL and ask for the price and delivery time. If it cannot read the page or quotes the wrong price, investigate.
6. Re-test after every security change. Agent access breaks silently when a WAF rule or platform default changes. Our Simulated Agent Buyer offers this as a regression run on every deploy or weekly (early access).

Regional differences

• United States. In Amazon v Perplexity, the Ninth Circuit vacated a preliminary injunction (4 August 2026) against the AI assistant in Perplexity's Comet browser, reasoning that "it is the user who 'accesses' Amazon's computers". The court said it did "not establish a new legal regime governing agentic AI", and noted that Amazon can still regulate access through its terms of service. Blocking remains your choice, and for most retailers the decision is commercial. Not legal advice.
• UK and EU. Consent banners and cookie walls are the most common interstitial that hides content from agents. The EU's text-and-data-mining opt-out is relevant to training-crawler choices (see the robots.txt notes above).
• Everywhere. An agent's IP address tells you where the vendor's servers are, not where the shopper is. Geo-blocking or geo-redirecting by IP can misroute agents in any market.

Common mistakes

• Fixing robots.txt and stopping there. The WAF decides access for user-triggered agents.
• Allowlisting by user-agent string. Anyone can send ChatGPT-User. Pair the token with the vendor's IP list, reverse DNS or a verified-bot signal.
• Blocking "AI" as one category. Training, search and user-triggered agents have very different commercial value.
• Blocking Google-Extended to hide from AI Overviews. Google says Google-Extended does not affect inclusion in Google Search; Googlebot rules and snippet controls govern AI features in Search.
• Hard-coding IP ranges. OpenAI, Perplexity, Anthropic and Apple publish IP lists, and those lists change. Fetch them automatically.
• Challenging the basket. A captcha on add-to-basket stops agents that browse for a shopper. Put your strongest controls on payment and account takeover, not on browsing.
• Copying publisher playbooks. Advice written for news sites about blocking AI often costs a retailer sales.

Sources

• OpenAI, Overview of OpenAI Crawlers, read 24 October 2026
• OpenAI Help Center, ChatGPT Work's Cloud browser allowlisting, read 24 October 2026
• Perplexity, Perplexity Crawlers, read 24 October 2026
• Anthropic, Claude Help Center: crawling and blocking, 7 April 2026
• Google, Google's common crawlers and AI features and your website, read 24 October 2026
• Microsoft, Overview of Bing crawlers and Verifying Bingbot
• Apple, About Applebot, 4 September 2026
• Cloudflare docs, Block AI Bots (1 July 2026), Verified bots (8 October 2026) and Web Bot Auth
• Akamai, Redefine trust with Web Bot Authentication, 10 November 2025
• Fastly docs, Managing bot traffic and Using system signals
• AWS, Bot Control rule group, AWS WAF Bot Control and Web Bot Auth announcement, 21 November 2025
• Shopify Help Center, Editing robots.txt.liquid, read 24 October 2026
• IETF, RFC 9309 and RFC 9421
• Amazon.com Services v Perplexity AI, No. 26-1444 (9th Cir., 4 August 2026)
• ConversionVector, Agent Commerce Index Q4 2026 (CC BY 4.0) and Agent Conversion Score specification v0.6