How does ChatGPT read my website?
Through two different user agents — one that indexes for search, one that fetches a page live when a user asks about it.
OpenAI operates separate crawlers for separate jobs. OAI-SearchBot builds the search index that backs ChatGPT search. ChatGPT-User fetches a specific page in the moment, when a user asks about a link or the assistant decides it needs to look. GPTBot is the one associated with collecting training data.
They are controlled independently in robots.txt. It is common — and often unintentional — to block GPTBot for training reasons and accidentally block the retrieval agents too, which removes you from answers entirely. Those are different decisions and should be made separately.
What actually gets read is the server-rendered HTML. Content that only appears after client-side JavaScript runs is unreliable at best. If your positioning statement is injected by a framework after hydration, assume it is not being seen.
You can check your own configuration with our AI Crawler Access Checker, which reads your robots.txt and compares live responses across crawler user agents.
See where you actually stand
32 checks, four pillars, one score — free and no account needed.
More in this category