Skip to content
Free score
Agent checkout

Page-fetching vs browsing agents: why 8 of 116 stores let one reach checkout, 96 the other

Why AI agents that fetch pages fail where browser agents succeed, what the Agent Commerce Index data shows, and how to make your store work for both.

By the ConversionVector teamUpdated 12 min read

Contents

Direct answer: AI shopping agents reach your store in two ways. Page-fetching agents read the raw HTML your server returns. Browsing agents drive a real browser like a person. In our Agent Commerce Index (Q4 2026), only 8 of 116 basket-tested stores (7%) let a page-fetching agent reach checkout, while 96 (83%) let a browsing agent through. The published data shows why. Most stores created the checkout and then refused the checkout page to anything that was not a real browser. Dozens more did not serve an add-to-basket form in their HTML at all. To work for both kinds of agent, put product facts and basket actions in the server-rendered page, and protect checkout without requiring a full browser.

This guide is for ecommerce leads and the developers who look after the storefront, CDN and checkout. It covers the two agent types, what the Index found store by store, why stores behave this way, the fixes in order and how to test both paths yourself.

Last verified: 26 October 2026 against the Agent Commerce Index Q4 2026 data file, our scoring specification and the bot documentation from OpenAI, Perplexity, Anthropic, Cloudflare, AWS and WooCommerce.

Two kinds of agent

Page-fetching agentBrowsing agent
How it worksRequests pages over HTTP and reads the HTML your server returnsDrives a real browser: runs JavaScript, keeps cookies, clicks and types
What it receivesOnly what is in the server responseThe page after scripts have run, as a person would see it
Typical useAnswering "what does this cost, is it in stock, what are the returns terms?"Completing a multi-step task such as filling a basket and checking out
Documented examplesOpenAI's ChatGPT-User: when users ask ChatGPT a question, "it may visit a web page" (OpenAI). Perplexity-User and Claude-User do the same for their assistants (Perplexity, Anthropic)ChatGPT Work's Cloud browser, which signs its requests with Web Bot Auth. OpenAI says that at launch it "cannot sign in to websites or complete payments" (OpenAI). Cloudflare's "Agent" category explicitly covers "chat fetch bots and browser-use agents" (Cloudflare)

Both matter commercially. The fetching agent decides whether an assistant can quote your price and stock accurately when a shopper asks. The browsing agent decides whether a shopper who delegates the purchase can actually complete it. Running a full browser is heavier than a plain request, which is why the fetch is what usually happens when an assistant looks something up.

What the Index tested

From the method notes on the Index page, for the Q4 2026 edition measured on 19 October 2026 (225 UK and US retailers, scored on spec v0.5):

  • The basket test added one in-stock product to an anonymous basket as ChatGPT's agent and opened the checkout page. Nothing was ever submitted.
  • A page-fetching agent is "ChatGPT's agent making plain page requests, without running a browser".
  • A browsing agent is "the same steps in a real browser".
  • Where the checkout turned the agent away, the same steps were repeated as an ordinary browser and, on Shopify, in a real browser.
  • "Browser only" marks a store whose checkout opened in a real browser but not to plain requests.
  • "Agents refused" marks a store that let the browser through but not the agent.
  • "Unverified" covers a checkout that refused every automated request, or a store that asked us to slow down (HTTP 429). These are left out of the basket figures.

The full method, including how the basket test works on each platform, is in our scoring specification under TRX-10. For the wider picture of the Index, see the Agent Commerce Index explained.

What happened, store by store

The headline figures are on the Index page. The breakdown below is our count of the per-store basket records in the published data file (CC BY 4.0), using its own outcome labels and notes. We re-counted it on 26 October 2026.

The 116 stores where the basket test ran

OutcomeStoresWhat the data says
Page-fetching agent reached checkout8"basket filled and the checkout page opened"
Reached checkout only as a browsing agent8881 of these: "the store created a checkout for the basket, but the checkout page was refused or challenged (403)"
Failed20Add to basket failed or was refused, product data was refused, or checkout did not open (for example "add to basket failed (500)" or "checkout did not open for a guest (200)")

A further 13 stores were "unverified" and left out: 7 because product data was rate limited (429), and 6 because the checkout was refused or challenged (403) for every automated request.

The 96 stores that could not be basket-tested

Reason in the dataStores
"no add-to-basket form in the page as served (it is probably added by script)"76
"custom Shopify storefront: the standard basket API is not served, so the basket test could not run"14
"no Shopify product handle in the sampled page address"6

Three findings from the records

1. Identity was not the problem in the basket test. The data file reports zero stores marked "agents refused". No checkout let an ordinary browser request through while turning away the request that identified as ChatGPT's agent. What separated the two agent types was how the request was made, not who claimed to make it.

2. The checkout page is where fetching agents stop. In 81 of the 88 browsing-only stores, the store accepted the basket and created a checkout, then returned a 403 or a challenge when the checkout page was requested without a real browser. All 88 were Shopify stores, which is also the only platform where the Index re-ran the steps in a real browser. Our interpretation is that browser-checking bot protection in front of the checkout page is the gate. The Index does not identify the mechanism. Shopify hosts the checkout and says network-layer bot management "is handled directly for stores on Shopify" (Shopify Help Center), so for most of these merchants it is not a setting they chose.

3. Many stores never offer a fetchable basket at all. For 76 stores, the add-to-basket form was not in the HTML as served, which suggests it is added by script. A page-fetching agent sees a product it cannot add. The 8 stores that did let a fetching agent through were on Salesforce Commerce Cloud (5), Magento (2) and one platform we could not identify. That is too small a group to say one platform is better, but it shows the route can be open.

Why stores behave like this

None of this is malicious. Each pattern has a reasonable origin:

  • Bot protection keyed on browser behaviour. Checkout is a target for card testing and other automated abuse, so bot protection there often looks for signs of a real browser, such as JavaScript execution and cookies. A plain HTTP client fails these checks by design, whatever it calls itself.
  • Client-rendered storefronts. Themes and headless front ends often inject the add-to-basket form, the variant picker or even the price with JavaScript. A person never notices. A fetching agent receives an empty shell.
  • Rate limits. A 429 protects origin capacity. Agents that make several requests in quick succession can trip limits set for human browsing.

The trade-off is real. The goal is not to remove protection, but to make sure it stops abuse rather than shopping.

What to fix, in order

1. Make product facts readable without a browser

  • Server-render the product name, price, currency, availability and variant options in the HTML.
  • Add complete Product and Offer JSON-LD on the server. Our product structured data guide has copy-ready examples.
  • Spec checks: LEG-08 (product name and price are in the HTML an agent receives) and OFF-02 (price present without a login, modal or region gate).

This fixes the most common commercial loss, where an assistant cannot quote you or quotes you wrongly. It is under your control on every platform.

2. Put a real add-to-basket action in the HTML

  • Render the add-to-basket <form> with the product or variant ID on the server, even if JavaScript enhances it later.
  • Give every variant its own URL and ID, so an agent can choose "navy, size L" without driving a JavaScript picker.
  • Where your platform has documented cart endpoints, keep them open to agent traffic. For example, WooCommerce's Store API provides "public Rest API endpoints for the development of customer-facing cart, checkout, and product functionality", including POST /wc/store/v1/cart/add-item, and needs no API key.
  • Spec checks: TRX-02 (cart is reachable by agent identities), TRX-09 (variants expressed structurally) and TRX-10 (an agent can put a product in the basket and reach checkout).

3. Tune checkout protection for verified agents

  • Allow agents that prove who they are. Cloudflare, Akamai and AWS WAF can verify Web Bot Auth signatures. Cloudflare uses them to verify bots and agents, and AWS WAF now allows verified Web Bot Auth agents by default in its AI bot category. Exempt verified agents from browser challenges on basket and checkout, and keep risk scoring on payment. Our guide on whether you are blocking AI shopping agents has the vendor settings.
  • Challenge on risk, not on every request. Place your strongest controls on payment submission and account login, not on viewing the basket or the first checkout step.
  • Hosted checkouts. If your platform hosts checkout (Shopify, for example), you may not control its bot protection. Test what agents receive. Where the platform offers agent channels or protocol checkout, those are the supported route for agent purchases.
  • Spec checks: ACC-12 (no JavaScript challenge or captcha interstitial served to agents) and TRX-03 (no captcha on cart or checkout pages).

4. Offer a machine route that does not need the page

Protocol checkouts (UCP, ACP) let an agent create a checkout session through an API instead of working through your pages. Where they are available for your platform and market, they sidestep the page-fetching problem. Availability differs by country and platform. Our overview of agentic commerce for retailers dates what is live where, and Google Universal Cart for retailers covers how Google hands a cart to your store over UCP. Spec checks: TRX-04 (UCP business profile), TRX-05 (ACP discovery or feed) and TRX-06 (MCP endpoint advertised).

How to test both paths yourself

Fetching path, in five minutes. The user-agent string below is the one OpenAI publishes for ChatGPT-User:

URL="https://www.example.com/products/example"
UA="Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot"
curl -s -A "$UA" -o page.html -w "status %{http_code}, %{size_download} bytes\n" "$URL"
grep -c 'application/ld+json' page.html                         # structured data present?
grep -Eio '<form[^>]*(cart|basket)[^>]*>' page.html | head -3    # add-to-basket form served?

Then send your platform's add-to-basket request and request the checkout page with the same client, keeping cookies (curl -c jar -b jar). A 403, a challenge page or an empty shell is what a fetching agent gets. Your laptop is not the agent's network, and a user-agent string alone is not a verified agent, so treat this as a first screen and check your CDN's security events for the definitive picture.

Browsing path. Ask a browser agent to buy a specific variant and note where it stalls. Our Simulated Agent Buyer runs browser agents through your purchase flow, records every step and reports the agent's reasoning at each stall.

Both, automatically. The free Agent Conversion Score reads your pages, cart and checkout as a real browser and as GPTBot, ChatGPT-User, PerplexityBot and ClaudeBot, so you can see where what an agent receives differs from what a browser receives.

Common mistakes

  • Testing only in your own browser. You pass every check a fetching agent fails.
  • Assuming the block is about AI. In the Index basket test, no store refused the agent by name. The checks that stopped agents were browser checks.
  • Rendering price with JavaScript "for performance". It costs you every assistant that reads HTML.
  • Treating a browsing-agent pass as job done. Browser agents can still be challenged, and fetching decides how you are quoted.
  • Removing checkout protection wholesale. Verify agents and challenge on risk. Do not open payment to everyone.

Limits of the data

  • One day of measurement (19 October 2026), and one agent identity (ChatGPT's agent) for the basket test.
  • The real-browser re-run was done on Shopify only, so the browsing figure is dominated by Shopify stores.
  • 96 stores could not be basket-tested. Their results are unknown, not failures.
  • The per-store breakdown above is our count of the published records. The Index page itself reports only the headline figures.

Sources

Questions

What is the difference between a page-fetching agent and a browsing agent?
A page-fetching agent requests your pages over plain HTTP and reads the HTML your server returns, without running JavaScript. A browsing agent drives a real browser, so it runs scripts, keeps cookies and clicks through the site like a person. In the Agent Commerce Index Q4 2026, 8 of 116 basket-tested stores let a page-fetching agent reach checkout, against 96 for a browsing agent.
Why did page-fetching agents fail so often?
In the published per-store data, the most common outcome was that the store created a checkout for the basket but refused or challenged the checkout page (HTTP 403) when it was requested without a real browser. That happened at 81 of the 88 stores reached only by the browsing agent. Separately, 76 stores could not be basket-tested because the page as served had no add-to-basket form.
Were stores blocking agents by name?
Not in the basket test. The Index marked no store as 'agents refused', meaning none let an ordinary browser request through while refusing the agent identity. The difference was how the request was made, not who claimed to make it.
Do I need to support page-fetching agents if browsing agents work?
It is worth it. OpenAI, Perplexity and Anthropic all document user-initiated fetchers that visit a page when someone asks a question, so fetching decides whether you are quoted correctly. Browsing agents can still be stopped by challenges. A store that works for both keeps every route to the sale open.
How do I test my store?
Fetch a product page without a browser and check that the name, price, stock and add-to-basket action are in the HTML. Then try adding to basket and opening checkout with plain requests, and check your CDN's security events. The free Agent Conversion Score reads your product pages, cart and checkout as a real browser and as four AI agents, so you can see where the two differ.

Published . Last updated . More in Agent checkout and the guide library.

Related guides