Does your website allow AI crawlers?
Search engines, AI chat assistants, and model-training pipelines each use their own crawlers, and sites can allow or block each one independently through robots.txt. This checker reads your robots.txt rules for a directory of known AI-related crawlers, then makes a handful of live requests to your homepage under different crawler identities to look for obvious blocking.
We check your public robots.txt and homepage. No login or form submission is performed. Genuine provider IP ranges are not tested.
How this check works
Robots policy. We fetch the site's robots.txt file and evaluate its rules against a curated directory of search crawlers, user-triggered fetchers, model-training crawlers, and content-use-control crawlers. A crawler with no explicit rule falls back to the wildcard User-agent: * group when one exists.
User-agent response testing. We request the public homepage once with a normal browser user agent and again with a small set of representative crawler user agents, then compare status codes to flag obvious user-agent-based blocking (403, 406, 429, or 5xx responses).
What this test cannot prove. These requests originate from Surfaced AI's infrastructure, not from the providers' published crawler IP ranges. A CDN or firewall may apply separate IP-based rules that this check will never see. A clean result means no blocking was detected by the checks we ran — it isn't a guarantee that every genuine crawler can reach every page.
This checks one signal. The full audit checks 32.
Run Free Audit