Nine AI crawlers visited oriontaurus.com 115 times between 30 August and 9 September 2026. A crawler is a program that reads web pages on its own, like Google’s. These nine belong to OpenAI, Anthropic, Meta, Amazon and others.
We recorded it ourselves, on our own server. You cannot check our table. You can build your own, and this article shows you how. The middle column below is the name you would write in robots.txt, a small text file on your website that tells crawlers what they may read.
| AI crawler | Name to use in robots.txt | Visits |
|---|---|---|
| Meta ExternalAgent | meta-externalagent | 57 |
| OpenAI GPTBot | GPTBot | 21 |
| Amazonbot | Amazonbot | 18 |
| Anthropic Claude-User | Claude-User | 8 |
| OpenAI SearchBot | OAI-SearchBot | 6 |
| PerplexityBot | PerplexityBot | 2 |
| Anthropic ClaudeBot | ClaudeBot | 1 |
| CommonCrawl CCBot | CCBot | 1 |
| ByteDance Bytespider | Bytespider | 1 |
| Total | 115 |
Our log ran for eleven days. The log stopped on 9 September and has not restarted, so the count is not current.
The number 115 is rough in both directions. Thirty-nine of those visits were requests for a real page we had written. Most of the rest were those same pages again under a second web address, plus sitemap files, feeds, and nine broken paths ending in “null” that look like a fault in our own recording. Our recorder also misses any page served from a cache, so some visits were never counted.
Does blocking GPTBot stop ChatGPT mentioning me?
Blocking GPTBot does not stop ChatGPT mentioning you. GPTBot is one of four OpenAI crawlers, and GPTBot is not the one that answers questions.
There are two kinds of AI crawler.
The first kind collects pages to help train a model. OpenAI says GPTBot collects “content that may be used in training our generative AI foundation models.” Anthropic says ClaudeBot collects “web content that could potentially contribute to their training.” Blocking a training crawler has no direct effect on whether an engine names you today. The model answering today finished training a long time ago.
The second kind asks for your page because a person has just asked a question. OpenAI’s search crawler is called OAI-SearchBot. OpenAI says sites “opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links.” Anthropic’s Claude-User does the same job. Blocking this second kind takes you out of answers people are asking for right now.
One company runs several crawlers under several names. OpenAI has four. Anthropic has three. A robots.txt rule naming GPTBot leaves the others alone, and two of OpenAI’s others are the ones deciding whether you appear in an answer today.
Some crawlers do both jobs. Meta says its crawler works “for use cases such as training foundation AI models or improving products by indexing content directly.” Two jobs, one sentence, and no way to tell which one visited you. Amazon runs three crawlers and says two never collect training material, so with Amazon the jobs can be separated. With Amazonbot, the one that visited us, they cannot.
We found no ByteDance documentation for Bytespider at all, so we record that Bytespider arrived and nothing about why.
Which AI crawler is actually hitting you the most?
The AI crawler visiting you most is probably not the one you have heard of. Meta’s crawler visited us 57 times. OpenAI’s GPTBot, the crawler everyone argues about, came second at 21.
When founders raise AI crawlers with us, they mean OpenAI. Sometimes Anthropic. We have not yet had one raise Meta.
One small site over eleven days proves nothing about the web. The crawler you assume is reading you may not be the crawler that is.
What pages do AI crawlers actually read on my site?
AI crawlers read your ordinary pages, and AI crawlers also ask for two files you have probably never opened. Our most requested page was /contact/, at nine visits. Next came /robots.txt and our sitemap, at eight each. A sitemap is a list of every page on your site, written for machines.
Counting every sitemap file, those two machine files account for 22 of the 115 visits, against 39 for pages we wrote. Roughly one request in five was a crawler asking what it may read and what exists.
The contact page coming first is worth knowing. When an engine is working out whether a business is real, the contact page carries the name, the location and a way to reach you. On the sites we open it is usually one of the thinnest. We have not counted that, so treat it as an impression.
We cannot tell you what order any of this happened in. Our log records the date and not the time. The first day in our log is the first day we started recording, which says nothing about what happened before. Crawlers also keep a copy of your robots.txt, and Amazon says its crawlers may “use a cached copy from the last 30 days.”
Why doesn’t Search Console or GA4 show any of this?
Neither Search Console nor GA4 shows AI crawlers, because neither tool was built to look for them. Search Console reports Google’s own crawling of your site and only Google’s. Search Console has no row for GPTBot.
GA4 cannot see AI crawlers at all. GA4 works by running a small piece of code inside a visitor’s web browser. A crawler asks for your page and never runs that code, so nothing is recorded.
How do I see the AI crawler hits on my own site?
AI crawler visits live in your server log, which is the file your website writes every time anyone asks for a page. If you can reach a command line on your server, this counts them:
sudo sh -c "zgrep -haiE 'GPTBot|OAI-SearchBot|ChatGPT-User|OAI-AdsBot|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User|meta-externalagent|meta-externalfetcher|Amazonbot|Amzn-SearchBot|Amzn-User|CCBot|Bytespider' /var/log/nginx/access.log*" | wc -lThree things about that line matter. The * sits inside the quotes, because outside them your own computer tries to use it before it has permission to read the folder, and on some servers that fails silently. The command counts whole lines instead of every match, because most crawler names appear twice in one line and counting matches roughly doubles the total. And the count runs slightly high if you publish pages with a crawler’s name in the address, because the name is in the line either way.
Your logs may only go back a day or two. Run ls -l on that folder to see how far back they go, because that is the real window you are reporting.
The command above only finds the names we gave it. The next command shows every visitor name your server recorded, including crawlers nobody has written about yet:
sudo sh -c "zgrep -ha -o '\"[^\"]*\"$' /var/log/nginx/access.log*" | sort | uniq -c | sort -rn | head -40If you cannot reach a command line, your hosting panel may hold the same files under “raw access logs”. WP Engine and Kinsta provide them. GoDaddy’s managed WordPress generally does not. Cloudflare reports AI crawler activity on every plan including free, if your site runs through Cloudflare.
One warning before you trust what you find. Anyone can make their traffic claim to be GPTBot, so treat the names in a log as claims rather than proof.
We built our own recorder, as a WordPress plugin. A WordPress plugin only sees visits that load WordPress, so anything served from a cache never reaches it. Our own 115 is therefore a floor as well as a rough number.
Should I block AI crawlers?
Blocking AI crawlers is a legitimate choice. Blocking them by accident is not, and by accident is usually how it happens.
A rule written years ago for a test site, a security plugin’s default list, a firewall switch someone flipped to stop copying: any of these can be deciding today whether ChatGPT’s search crawler may read you. Nobody in the business knows the rule is there. We open enough of these to expect one, though we have not counted them.
So decide on purpose, and decide about training separately from answering. Then read your robots.txt back and check it says what you think it says.
One thing to know before you rely on it. Robots.txt is a request. Robots.txt is not a lock, and four of these companies say so themselves about the crawler that fetches a page when a person asks a question.
- Perplexity: “Since a user requested the fetch, this fetcher generally ignores robots.txt rules.”
- Amazon, on Amzn-User: “Because actions taken by Amzn-User can be initiated by a user, it may not follow all robots.txt directives.”
- Meta, on its user-request crawler, which did not visit us: it “may bypass robots.txt rules.”
- OpenAI, on ChatGPT-User: “Because these actions are initiated by a user, robots.txt rules may not apply.”
Anthropic is the one that does not say this. Anthropic says its crawlers “respect ‘do not crawl’ signals by honoring industry standard directives in robots.txt.” We found no ByteDance documentation either way.
Does a crawler hit mean ChatGPT will mention me?
No. A crawler visiting means an engine can reach you, and reaching you is one condition out of several.
Reaching you is also the only condition you can prove from your own records rather than a vendor’s dashboard, which is why we start there. A crawler arriving still tells you nothing about whether the engine worked out what your business is.
We are Orion & Taurus. Alex and Jenn Hays, two people. We work on whether answer engines can reach your business, describe it correctly, and name it when a buyer asks who does what you do. Reading your crawler log is where we start.
Run the first command. You may find you do not need us for this part.
Sources
Every figure and quotation, with the date it was checked. Operator pages were re-checked on 21 September 2026.
Our own server. AI crawler visits to oriontaurus.com, 30 August to 9 September 2026, eleven days. Nine crawlers, 115 recorded visits. Recorded by ot-ai-crawler-log, our own WordPress plugin, aggregated 19 September 2026. The record stops on 9 September 2026. It stores a date and no time, so it cannot show what order anything arrived in. It only sees requests that load WordPress, so cached and edge-served visits are missing. A reader cannot reopen a private server record, which is why the article gives the method instead.
Per-crawler counts. Meta ExternalAgent 57, GPTBot 21, Amazonbot 18, Claude-User 8, OpenAI SearchBot 6, PerplexityBot 2, ClaudeBot 1, CCBot 1, Bytespider 1.
What the 115 rows are. Distinct content pages 39, raw ?p= permalinks 21 (the same pages already counted under their ordinary addresses), sitemap XML 13, feeds 13, /embed/ endpoints 9, paths ending /null 9, /robots.txt 8, homepage with a tracking parameter 1, /wp-sitemap.xsl 1, address carrying a LinkedIn share parameter 1. Robots.txt plus all sitemap files is 22.
Most-requested paths. /contact/ 9, /robots.txt 8, /wp-sitemap.xml 8, /why-us/ 5, /contact 4, then seven paths at 3 each.
Crawler names. Every robots.txt name in this article comes from the operator documentation below, including its capitalisation. One exception: Bytespider is the name as our own log recorded it, and we could not confirm it against any ByteDance page.
GPTBot, OAI-SearchBot, ChatGPT-User and OAI-AdsBot: purposes, names, the navigational-links clause and the ChatGPT-User robots.txt clause. OpenAI, https://developers.openai.com/api/docs/bots, checked 21 September 2026.
ClaudeBot and Claude-User purposes, and Anthropic’s statement that its bots “respect ‘do not crawl’ signals by honoring industry standard directives in robots.txt.” Anthropic, https://support.claude.com/en/articles/8896518, checked 21 September 2026.
PerplexityBot’s purpose and Perplexity-User’s statement that “since a user requested the fetch, this fetcher generally ignores robots.txt rules.” Perplexity, https://docs.perplexity.ai/guides/bots, checked 21 September 2026.
Meta ExternalAgent’s two stated purposes, and meta-externalfetcher’s statement that it “may bypass robots.txt rules.” Meta, https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/, checked 21 September 2026. Meta-externalfetcher does not appear in our log.
Amazonbot, Amzn-SearchBot and Amzn-User: purposes, the two statements that they “do not crawl content for generative AI model training,” the Amzn-User robots.txt clause, and the statement that Amazon’s crawlers may “use a cached copy from the last 30 days.” Amazon, https://developer.amazon.com/amazonbot, checked 21 September 2026.
Bytespider. No ByteDance documentation of the crawler’s purpose, name or robots.txt handling was found on 19 or 21 September 2026.
Search Console reports Google’s own crawling. The Crawl stats report covers “Google’s crawling history on your website,” broken down by Googlebot type, with an “Other agent type” row for “[a]nother Google crawler not specified here.” Google, https://support.google.com/webmasters/answer/9679690, checked 21 September 2026. The bracketed letter marks a capital lowercased to fit the sentence.
GA4 cannot see crawlers. This rests on the mechanism alone: GA4 collects events from JavaScript executed in a browser, and a crawler requesting a page does not execute it. Google also excludes known bots automatically and states that “at this time, you cannot disable known bot traffic exclusion or see how much known bot traffic was excluded,” but does not say whether these nine are on that list, so the article does not rely on it. https://support.google.com/analytics/answer/9888366, checked 21 September 2026.
Cloudflare AI Crawl Control, available on all plans, requires the domain to be proxying traffic through Cloudflare. https://developers.cloudflare.com/ai-crawl-control/, checked 21 September 2026.
Log rotation and permissions. Ubuntu’s nginx and apache2 packages rotate daily with delaycompress, so yesterday’s file is plain text. RHEL rotates weekly with rotate 4 and ships compression commented out. RHEL’s /var/log/httpd is 0700 root:root and Ubuntu’s /var/log/apache2 is 0750 root:adm, which is why the * has to sit inside the quotes. Verified by reproduction on 21 September 2026.
Double-counting. A test log built from real crawler user-agent strings, counted by matches rather than lines, returned 6 for GPTBot, 4 for PerplexityBot and 4 for ClaudeBot where the true line counts were 2, 2 and 2. Reproduced 21 September 2026.
Checking a crawler is who it says it is. Most operators publish their address ranges: OpenAI at openai.com/gptbot.json and three companion files, Anthropic at claude.com/crawling/bots.json, Perplexity at www.perplexity.com/perplexitybot.json and perplexity-user.json, Amazon on three pages under developer.amazon.com/amazonbot, Common Crawl at index.commoncrawl.org/ccbot.json. Only Common Crawl documents a check, warning that “we are aware of crawlers falsely identifying themselves as CCBot” and supporting reverse DNS. All checked 21 September 2026. The article says only that a name in a log is a claim, which is all these pages support.
