Surfaced AI

Should I block AI crawlers in robots.txt?

Decide per crawler — blocking training crawlers is a legitimate choice, but blocking retrieval agents removes you from answers.

The important distinction is training versus retrieval. Training crawlers such as GPTBot, ClaudeBot, and Google-Extended collect data used to build models. Retrieval agents such as OAI-SearchBot, PerplexityBot, ChatGPT-User, and Perplexity-User fetch pages to answer a question now.

Blocking training crawlers is a reasonable content decision with no direct effect on whether you are recommended today. Blocking retrieval agents makes you uncitable — which is usually the opposite of what a founder wants.

The frequent accident is a blanket `User-agent: * / Disallow: /` rule, or a CDN bot filter that rejects unfamiliar agents, taking out both categories at once.

Audit what you actually serve rather than what you think you serve — our AI Crawler Access Checker compares live responses across agents.

See where you actually stand

32 checks, four pillars, one score — free and no account needed.

Run free audit

More in this category

How do AI crawlers discover my website?Through links, sitemaps, and search indexes — much the same way traditional crawlers do.Can AI crawlers read JavaScript-rendered content?Often not. Assume anything that requires client-side rendering may be missed entirely.Does site speed affect AI visibility?Indirectly. Speed itself is not scored, but a page that times out during a fetch simply is not read.Could my CDN or firewall be blocking AI crawlers?Very often, yes — bot-protection rules commonly reject unfamiliar user agents without anything appearing in robots.txt.
All answers

Your competitors are already being recommended. Catch up.

Run the audit. Decide after. Understand why AI engines recommend competitors instead of you. Free, in about two minutes. No card, no signup, no obligation.

Or browse the sample audit first