On this page
The short version
Does ChatGPT read your website? You can check in four steps, from quickest to most certain:
- robots.txt: do you allow OAI-SearchBot, the crawler behind ChatGPT search? (Check 1)
- Your firewall or CDN: services like Cloudflare can turn AI crawlers away before robots.txt is even read. (Check 2)
- Your server logs: do the crawlers actually visit, and are they real? (Check 3)
- Your citations: does ChatGPT actually link to your pages in its answers? This is the one that matters. (Check 4)
We ran the first two checks on five real business websites. All five let ChatGPT in. Yet AI cited one of them in 1,471 answers over four weeks, and another in fewer than five. Being readable is the starting line, not the finish.
ChatGPT's three visitors
The short answer: OpenAI sends three different agents, and blocking each one does something different.
| Agent | What it's for | Does robots.txt apply? | If you block it |
|---|---|---|---|
| OAI-SearchBot | ChatGPT's search answers | Yes | You won't be shown in ChatGPT search answers (you can still appear as a plain navigational link) |
| ChatGPT-User | Opening a page while ChatGPT answers someone | May not, because a user started it | It may make little difference |
| GPTBot | Collecting content to train future models | Yes | You opt out of training only, not ChatGPT search |
That's from OpenAI's own overview of its crawlers. Two points people often get wrong:
- Blocking GPTBot doesn't hide you from ChatGPT search. Lots of sites block GPTBot to keep out of training, which is a fair choice, and still appear in ChatGPT's answers.
- Changes aren't instant. OpenAI says it can take about 24 hours after a robots.txt change for search to adjust.
Check 1: your robots.txt
The short answer: open yourdomain.com/robots.txt and look for any rule that names OAI-SearchBot, or blocks everyone.
Your robots.txt is a plain text file at the root of your site. Each crawler follows the block with its own name, or the "User-agent: *" block if nothing names it. Google's introduction to robots.txt explains the format.
| You want… | Lines in robots.txt |
|---|---|
| ChatGPT search, but no training | User-agent: GPTBot → Disallow: / (and no rule blocking OAI-SearchBot) |
| Everything open (the default) | No rules naming OpenAI's agents, and no "Disallow: /" under User-agent: * |
| Keep out of ChatGPT search too | User-agent: OAI-SearchBot → Disallow: / |
Watch for one common accident: a "Disallow: /" under "User-agent: *", left over from a staging site. That blocks every crawler that follows robots.txt, OpenAI's included.
Check 2: your firewall and Cloudflare
The short answer: a firewall can turn an AI crawler away even when robots.txt says yes, and you'd never see it in robots.txt.
Robots.txt is a polite request. A firewall or CDN is a locked door. In July 2025, Cloudflare, which sits in front of a large share of websites, announced it would block AI crawlers by default for new customers, with a setting to change it. It has since refined which kinds of AI crawler its defaults block, so check the current settings in your own dashboard rather than assuming.
Our crawler check found exactly this kind of block on one of our five sites:
| Website | robots.txt | Live request with each crawler's name |
|---|---|---|
| Kato Electrical | All allowed | All got through |
| Dental training institute | All allowed | GPTBot turned away (status 429); all others got through |
| Industrial software company | All allowed | All got through |
| Online mobility store | All allowed | All got through |
| Shower door company | All allowed | All got through |
The dental institute's robots.txt didn't block GPTBot, but its firewall did. Because GPTBot is only the training crawler, that didn't stop ChatGPT search reading the site. In fact, the institute's site was cited more than any other business website in our data.
How to check yours:
- Log in to your CDN or hosting firewall (for Cloudflare, look in your domain's security and bot settings).
- Find any setting that blocks "AI bots" or "AI crawlers".
- Make sure it allows search crawlers like OAI-SearchBot, unless you mean to block them.
Check 3: your server logs
The short answer: look for visits from OAI-SearchBot and ChatGPT-User, and check they really come from OpenAI.
- Find the visits. If you can see raw server logs (most hosting control panels show them), search for "OAI-SearchBot", "ChatGPT-User" and "GPTBot" in the user agent. Regular OAI-SearchBot visits mean ChatGPT search knows your site exists.
- Check they're genuine. Anyone can pretend to be GPTBot. OpenAI publishes the IP addresses its agents use: searchbot.json, gptbot.json and chatgpt-user.json. A request using OpenAI's name from an address outside those lists isn't OpenAI.
- Don't panic about quiet logs. A small site may only see occasional visits. ChatGPT-User only comes when someone's conversation needs your page.
If you can't see server logs (many website builders hide them), skip to check 4. It tells you more anyway.
Check 4: does ChatGPT cite you?
The short answer: the real test is whether ChatGPT links to your pages in its answers, and our data shows that being allowed in doesn't guarantee it.
Our tracker, LLM Visibility Lens, asks each business's buying questions every day in ChatGPT and four other AI systems, and records every link. From 30 September to 28 October 2026, the five sites above were cited:
| Website | AI answers citing its own site (five AI systems) |
|---|---|
| Dental training institute | 1,471 |
| Industrial software company | 258 |
| Kato Electrical | 247 |
| Online mobility store | 57 |
| Shower door company | fewer than 5 |
Every one of them let ChatGPT in, yet the citations ranged from fewer than 5 to 1,471. Looking at ChatGPT alone, from 30 September to 20 October it linked to the dental institute's site in 145 answers, a Massachusetts roofer's in 35, and Kato's in 8.
So what makes the difference, once the door is open? From our other studies:
- What's on the pages. Pages that name the business, the service and the place get read and named; see which pages AI reads. Our crawler check flagged that Kato's home page has his name only in the title, not in the opening lines.
- What the rest of the web says. AI reads directories, lists and competitors' sites far more than any one business's site; see where AI gets its answers.
- Whether a citation comes with your name. When ChatGPT does cite your site, it usually names you, but not always; see when your own web page helps a competitor and which websites ChatGPT trusts.
You can run check 4 by hand: ask ChatGPT your customers' buying questions on several days, and note whether it links to your site.
What to allow and what to block
The short answer: allow the search crawlers, decide on training for yourself, and don't block by accident.
- Allow OAI-SearchBot if you want to appear in ChatGPT search. For a business that wants customers, that's almost always yes.
- Decide on GPTBot. Blocking it keeps your content out of training without hurting ChatGPT search. It's a business choice, not a visibility one.
- Leave ChatGPT-User alone. It acts when a real person asks ChatGPT about your page.
- Check your firewall after any security change. That's where accidental blocks hide.
- Submit your sitemap to Bing Webmaster Tools and keep pages fast and readable without heavy scripts, so every crawler gets the same clear content.
Other AI crawlers worth knowing
Our crawler check tests these too, because other AI systems use their own agents:
- PerplexityBot: Perplexity's answers.
- Googlebot: Google Search, including AI Overviews and AI Mode. Blocking it removes you from Google altogether.
- Google-Extended: a robots.txt setting for Gemini, separate from Google Search. It's a rule, not a separate visitor, so there's no live request to test.
- Bingbot: Bing and Microsoft Copilot.
- ClaudeBot: Anthropic's Claude.
Each one follows its own block in robots.txt, so check them one by one. Compare how differently the AI systems use what they read in which AI recommends local businesses most.
How to check yours in one go
LLM Visibility Lens runs this AI crawler check on every plan, including the free one. It tests robots.txt and a live request for each crawler, then checks whether your key pages make your name, service and place clear. It also tracks whether ChatGPT and four other AI systems actually cite and name you, every day. See Kato's in the live demo, try the free question finder, or compare plans. Agencies can run it for every client; see the agency plan. For the full list of fixes, start with the AI visibility checklist.
More reading:
- How to get recommended by ChatGPT
- How to rank in ChatGPT
- ChatGPT citations for local businesses
- What to fix first when AI ignores a client
- AI share of voice
- The directories AI reads
- Questions nobody searches on Google
- Invisible on Google, recommended by AI
- Why a single AI visibility score can mislead you
- How to report AI visibility to clients
- 5 AIs, same questions, 1,814 answers
- 5 AI systems, 35 questions
- Google Business Profile and AI answers
- Google AI Mode for local businesses
- How to rank in AI Overviews
- AI Overview tracking for near me searches
- Perplexity vs ChatGPT
- Spoken questions
- Why there's no rank #1 in ChatGPT
- What one free check can and can't do
- We asked AI who to hire 80 times
The method is on the how it works page. There's also a comparison with Semrush's AI visibility tool, the changelog, the FAQ page and the security page. If you'd like me to check whether ChatGPT can read your site, get in touch.
Questions people ask
Does ChatGPT read my website?
It can if your robots.txt and firewall let OpenAI's search crawler, OAI-SearchBot, in. The best proof is whether ChatGPT cites your pages in its answers. In our data, five sites all let ChatGPT in, yet AI cited them anywhere from fewer than 5 to 1,471 times in four weeks.
What is the difference between GPTBot and OAI-SearchBot?
GPTBot collects content that may be used to train OpenAI's future models. OAI-SearchBot finds pages for ChatGPT's search answers. Blocking GPTBot opts you out of training but doesn't remove you from ChatGPT search; blocking OAI-SearchBot does.
Should I block GPTBot?
It's a business choice. Blocking GPTBot keeps your content out of OpenAI's model training and doesn't affect ChatGPT search. One site in our data had GPTBot blocked by its firewall and was still the most-cited business website we tracked.
Can Cloudflare stop ChatGPT reading my site?
Yes. A firewall or CDN can turn AI crawlers away before robots.txt is even read. Cloudflare announced default blocking of AI crawlers for new customers in July 2025, so check your bot settings and make sure search crawlers are allowed if you want to appear in AI answers.
How do I know if a visit from GPTBot is real?
Check the visitor's IP address against the lists OpenAI publishes for each agent: searchbot.json, gptbot.json and chatgpt-user.json. A request using OpenAI's name from an address outside those lists isn't from OpenAI.
How long does it take ChatGPT to notice a robots.txt change?
OpenAI says it can take about 24 hours for its search systems to adjust after a robots.txt update. Whether ChatGPT then cites your pages depends on what's on them and what the rest of the web says about you.
Harjeet Singh Bedi Founder, LLM Visibility Lens · SEO consultant, Bedi SEO
Harjeet has worked in SEO since 2010, first in agencies and now running Bedi SEO for local businesses, online stores and agencies. He built LLM Visibility Lens after clients kept asking whether they showed up in ChatGPT and other AI tools and nobody could give them an honest, measured answer. Every figure in this article comes from answers his tool collected.