AI Crawler & Bot Directory
The Story
Directory of all 28 AI crawlers — GPTBot, ClaudeBot, PerplexityBot, Google-Extended and more — with robots.txt rules to allow or block each one.
tools 19 guides 41 bots 28 about Loading Built for the era when robots read your website.
Free micro-tools for getting found, read, and cited by ChatGPT, Claude, Perplexity & Google AI Overviews.
No accounts. No paywalls. Most tools run right in your browser.
This is a maintained directory of the AI bots and user agents that crawl the web — the crawlers behind ChatGPT, Claude, Perplexity, Gemini, and the training datasets feeding every major model. Each profile explains what the bot does, whether it actually respects robots.txt, and gives copy-paste rules to allow or block it. To check what your own robots.txt currently permits, run the AI Crawler Access Checker .
Overview
Directory of all 28 AI crawlers — GPTBot, ClaudeBot, PerplexityBot, Google-Extended and more — with robots.txt rules to allow or block each one.
Directory of all 28 AI crawlers — GPTBot, ClaudeBot, PerplexityBot, Google-Extended and more — with robots.txt rules to allow or block each one.
tools 19 guides 41 bots 28 about Loading Built for the era when robots read your website.
Free micro-tools for getting found, read, and cited by ChatGPT, Claude, Perplexity & Google AI Overviews.
No accounts. No paywalls. Most tools run right in your browser.
FAQ
What is AI Crawler & Bot Directory?
Directory of all 28 AI crawlers — GPTBot, ClaudeBot, PerplexityBot, Google-Extended and more — with robots.txt rules to allow or block each one.
Where can I access AI Crawler & Bot Directory?
Use the official product link below. It is the destination recorded for this listing.
Where should I check current pricing and availability?
Check the current product page before making a decision. The directory's pricing category is a listing field, not a live price quote or availability guarantee.
What stack is AI Crawler & Bot Directory built with?
The CompassPad entry does not list a specific stack for AI Crawler & Bot Directory.
Is AI Crawler & Bot Directory free, or does it cost money?
The CompassPad listing records AI Crawler & Bot Directory as free to use. Check the product page for any optional paid options.
What we checked
- DISCOVEREDFound in the wild
This listing was discovered from a public launch post and the description above is drawn from the product's own website.
- LIVEProduct site
The product site resolved when this entry was added to the catalog.
Discussion
No comments yet. Be the first.
Join the conversation. Sign up to comment.
Sign up free