Surfaced AI

Could my CDN or firewall be blocking AI crawlers?

Very often, yes — bot-protection rules commonly reject unfamiliar user agents without anything appearing in robots.txt.

This is one of the most common invisible causes. Your robots.txt can be permissive while a WAF rule, bot-management setting, or rate limiter quietly returns 403 to anything that does not look like a browser.

Because it happens at the edge, nothing in your application logs or site config hints at it. The site looks fine to you and is unreachable to them.

Test by requesting your own homepage with a crawler user agent and comparing the status code to a normal browser request. A 403, 406, or 429 is the tell.

Our AI Crawler Access Checker does exactly this comparison across several agents, which is usually the fastest way to confirm or rule it out.

See where you actually stand

32 checks, four pillars, one score — free and no account needed.

Run free audit

More in this category

How do AI crawlers discover my website?Through links, sitemaps, and search indexes — much the same way traditional crawlers do.Can AI crawlers read JavaScript-rendered content?Often not. Assume anything that requires client-side rendering may be missed entirely.Should I block AI crawlers in robots.txt?Decide per crawler — blocking training crawlers is a legitimate choice, but blocking retrieval agents removes you from answers.Does site speed affect AI visibility?Indirectly. Speed itself is not scored, but a page that times out during a fetch simply is not read.
All answers

Your competitors are already being recommended. Catch up.

Run the audit. Decide after. Understand why AI engines recommend competitors instead of you. Free, in about two minutes. No card, no signup, no obligation.

Or browse the sample audit first