websiteaudit.in

Free tools / AI Crawler Checker

AI crawler checker

Enter a website to see whether AI assistants can read it. We check the site's robots.txt rules for each AI bot, separating search bots (which decide whether ChatGPT, Claude and Perplexity can find and cite you) from training bots (which only collect data for training models). Blocking training bots is a fair choice; blocking search bots means AI assistants can't recommend you.

Questions

What's the difference between AI search bots and training bots?

Search bots, like OAI-SearchBot, Claude-SearchBot and PerplexityBot, fetch pages so the assistant can find and cite them in answers. Training bots, like GPTBot, ClaudeBot and Google-Extended, collect content to train future models. You can block training while still allowing search.

How do I block AI training but allow AI search?

In robots.txt, add a group for each training bot with "Disallow: /" (for example User-agent: GPTBot) and leave the search bots allowed. The vendors document their bot names, and they say they respect these rules.

My robots.txt allows AI bots. Why might they still be blocked?

Some hosts and CDNs block AI crawlers at the firewall, before robots.txt is ever read. Cloudflare, for example, has a setting to block AI bots. If you use one, check that setting too.

What is llms.txt?

An optional, emerging file that gives AI tools a plain-text summary of your site. It's quick to add, but major AI search engines don't rely on it yet, so it's a nice-to-have rather than a priority.

Want the whole picture?

This checks one thing. The full audit runs every check on speed, SEO, mobile and security, adds AI readiness, ranks the fixes by impact and gives you a copy-paste AI prompt for each one. Free, no signup.

More free tools