Are you accidentally invisible? Auditing CDN and firewall bot blocking
Last reviewed
Short answer
Robots.txt is only the polite gate. A CDN, firewall or bot-protection product can block AI crawlers your robots.txt allows, silently, with no entry in any report you read. The audit: read your rendered robots.txt, check every security layer you own for bot rules, and verify from logs that AI crawlers actually reach your pages.
Why this failure is silent
A blocked crawler does not file a complaint. It gets a 403, tries less often, and your store simply stops appearing in the answers that crawler feeds. Nothing in Shopify admin reports it, Search Console does not cover AI bots, and the assistant keeps answering shopper questions about your products, just without reading your pages.
Security products make this worse by being right most of the time. Bot protection that blocks scrapers is doing its job, and AI crawlers look like scrapers to a rule written before 2024. The audit is about finding where a good rule has a bad side effect.
Layer one: what your robots.txt actually says
Start at yourstore.com/robots.txt, the rendered file, not any template. Read every User-agent group against the reference table and note anything touching OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-SearchBot or Claude-User.
On a stock Shopify setup with no custom template, this layer is fine and costs one minute to confirm. The point of checking first is that everything downstream is only worth auditing once you know the polite gate is open.
Layer two: every proxy you own
This layer only exists if traffic passes through infrastructure you control: your own Cloudflare zone in front of a landing page, blog or headless storefront, a WAF, or any bot-protection product. A default Shopify store with DNS-only records has no layer two, which is worth saying because it exempts most merchants.
On Cloudflare, the ground truth lives in AI Crawl Control, available on all plans: the Crawlers tab lists each AI crawler with its category and an Allow or Block action, and the metrics show whether requests are succeeding. Check Security settings for the legacy Block AI bots toggle, and read your WAF custom rules, because AI Crawl Control implements blocks as a WAF rule and your own older rules act independently of it. The September 15 default change raises the stakes on exactly this audit.
Layer three: proof from the outside
Configuration says what should happen. Verification is watching what does. The cheap test is fetching your own product page with each bot's user-agent string and comparing status codes: a 200 for a browser string and a 403 for ChatGPT-User is a finding. It is not perfect proof, since sophisticated protections check more than the string, but it catches the common cases.
The strong test is logs, where you have them: CDN logs or server logs for a headless front, filtered to the published user agents. Bot hits on your product pages are the leading indicator that the pipeline works, and their absence over weeks, while competitors get cited, is the symptom this page exists to explain. On hosted Shopify you do not get log access, which makes the outside-in tests and sampled visibility measurement the available instruments.
- curl -A "OAI-SearchBot" https://yourstore.com/products/your-product, compare to a normal fetch
- Repeat for ChatGPT-User, PerplexityBot, Perplexity-User, Claude-User
- A 403, 429 or challenge page for bots only means a layer is blocking
- Where logs exist, filter for the published agent strings and watch product page hits
What to do with a finding
Fix the narrowest layer that produced it. A robots.txt rule: edit the template. A Cloudflare crawler action: one click to Allow in the Crawlers tab. A WAF custom rule: add an exception for the specific verified bots rather than loosening the rule wholesale. Then re-run the same test that found the problem, because a fix you did not re-test is a hope.
Give it a day, since OpenAI states robots.txt changes take about 24 hours to register, and then watch the trend rather than a single answer. Silent blocks build slowly and clear slowly.
Frequently asked questions
I'm on plain Shopify with no Cloudflare account. Can I skip this?
Mostly. Confirm your rendered robots.txt is clean and you are done, because there is no security layer of yours between crawlers and your store. The audit exists for merchants running their own proxy, WAF or bot protection in front of anything.
Can Shopify itself be blocking AI bots?
Shopify manages its own edge and does not publish per-bot blocking for merchant storefronts. What you control and can audit is your robots.txt and any infrastructure you put in front. Product data additionally reaches AI surfaces through Catalog, which does not depend on crawling at all.
How often should I re-run this?
When you change security settings, when a new protection product arrives, and on vendor default changes like Cloudflare's September 15 one. Between those events, a quarterly re-check alongside your other re-verification is proportionate.
My curl test shows a challenge page for everything. Is that a block?
For a bot, effectively yes. Crawlers do not solve challenges. If every request from a bot user-agent hits a challenge, whatever issues those challenges is your layer to add exceptions to, using the operator's published IP ranges to scope them safely.
Sources
Keep reading
Score your own catalog
Entitled grades every product against the rules on this page, explains each failure, and fixes them with AI you approve. The score is free forever.