How to Track GPTBot, ClaudeBot, and PerplexityBot Hits on Your Site

Crawler logs tell you which agents request your pages and whether those requests succeed. They do not tell you whether an AI answer recommends your brand. Use them to diagnose access problems, then measure citations and customer traffic separately.

This guide covers the monitoring workflow. For the actual access rules, read our robots.txt guide for AI search and training.

Separate search, training and user requests

Group requests by purpose before comparing counts. OAI-SearchBot and PerplexityBot support search discovery. GPTBot is associated with OpenAI training crawls. ChatGPT-User and Perplexity-User represent user-triggered visits rather than automatic search indexing.

OpenAI documents independent search and training controls in its crawler reference. Perplexity explains its search crawler and user fetcher in its bot documentation. Consult the current provider documentation for any other agent you track, including ClaudeBot.

Google-Extended is a robots.txt product token, not a separate HTTP user-agent. Do not expect to identify visits bearing that name in your access logs. Its documented purposes include specified Gemini training and grounding uses; it does not control inclusion in Google Search. See Google's crawler reference.

Choose a source of request data

Server and edge logs can show the requested URL, response status, timestamp and user-agent. Confirm your retention period and whether cached requests are visible. Application logs alone can miss requests handled or blocked at the CDN.

Hosting and CDN bot reports can help investigate requests rejected before they reach the application. Check which dimensions, identity verification and export options your provider actually exposes.

Application tracking can bring the requests that reach your application into a shared dashboard. Aeranko's crawler view supports reviewing recorded visits by crawler, page and status. Use edge logs alongside it when investigating firewall blocks.

Choose the source that answers your immediate question. If a crawler gets challenged at the edge, adding an application tracker will not make that blocked request appear in application logs.

Record four useful signals

1. Identity and purpose. Record the claimed agent and whether you have verified it using the provider's guidance. Anyone can send a familiar user-agent string.
2. Page coverage. Compare visits to important public URLs, including pricing, product information and documentation. Missing visits are a reason to investigate, not proof of a block.
3. Response quality. Inspect errors, redirects and response bodies. A successful status containing a challenge page is not successful content access.
4. Changes over time. Annotate deployments and firewall changes. Compare patterns over a consistent period rather than treating one quiet hour as a regression.

Keep training-bot counts out of your search-crawler totals. An increase in training requests is not evidence that search answers cite you more often.

Investigate a suspected block

Check the deployed robots.txt file for the exact agent and URL. A rule blocking an internal path does not block the homepage. Conversely, an agent-specific group may need its own exclusions rather than inheriting the wildcard group.

Then inspect CDN challenges, rate limits, authentication redirects and server errors. Reproduce the response where possible and confirm that public content is present. For a configuration example and a complete checklist, use the AI crawler access guide.

Turn findings into a useful report

Write a concrete observation: “Requests claiming to be OAI-SearchBot received a challenge on the pricing page after Tuesday's firewall change.” Include the URL, time window and verification status. Assign an owner and check subsequent requests after a fix.

Avoid presenting crawler growth as revenue or citation growth. Pair request data with AI visibility measurements and AI referral traffic to see whether technical improvements are followed by customer discovery.

Start with your public pages

Choose a small set of pages that explain what you sell, who it serves and how to get started. Review a week of request data, investigate failures and record what you changed.

You can run a free Aeranko audit without installing a tracker. It provides a starting point for technical signals and available visibility measurements; request logs remain the place to investigate individual crawler visits.