AI Crawler
Checker.
Paste your robots.txt and a path, and see exactly which AI crawlers — GPTBot, ClaudeBot, PerplexityBot, Google-Extended and more — can read it, following the same group and path rules a compliant bot would. Free, instant, and it never leaves your browser.
Paste it.
Pick a path.
Paste your robots.txt content, choose the path to test — "/" by default — and see the result for every bot in the list below, with the rule that decided it.
Access for /
| Bot | Operator | Status | Matched rule |
|---|
Bot list last reviewed 2026-11-02. Operators add and rename user-agents without notice — re-check against their own documentation if this result matters for a decision.
We read the rules.
We don't crawl you.
Most "robots.txt checkers" online fetch a live domain, which means sending your URL to someone else's server and trusting their parser. This tool skips that step entirely: you paste the text yourself, everything runs in a script in your own browser, and nothing about your site — not the domain, not the file, not the path you're curious about — goes anywhere else. That matters for a file that often encodes exactly which parts of a business you'd rather not crawl: unreleased sections, staging paths, admin areas.
It also means the tool works just as well on a robots.txt that isn't live yet — a draft you're about to deploy, a client's file you're reviewing before a migration, or a competitor's file you copied into a text editor to study. Paste it, pick a path, and see how every major AI crawler would read it today, under the actual IETF RFC 9309 rules rather than a guess.
What this checker
actually evaluates.
robots.txt files are organised into groups, each one starting with one or more User-agent lines followed by Allow and Disallow rules. For every bot in the list, the tool finds the most specific group that names it — falling back to the general * group when no specific one exists — then, within that group, applies the rule with the longest matching path. If two rules match a path equally, Allow wins over Disallow; if nothing matches at all, the path is allowed by default. That's the whole of RFC 9309's matching logic, and it's exactly what this tool implements, with no shortcuts.
It checks 14 named AI crawlers — training bots like GPTBot and ClaudeBot, search-and-retrieval bots like OAI-SearchBot and PerplexityBot, and a few whose purpose sits in between, like Google-Extended and CCBot. What it does not do: verify that a given bot actually honours robots.txt in practice (some don't), check whether your server or CDN enforces anything beyond the file, or fetch a live domain — see the disclaimer above. Those are different questions, covered in the AI crawler control guide, not in this tool.
robots.txt, llms.txt
and Content-Signal.
These three get conflated constantly, and the confusion leads to bad decisions. robots.txt is the only one of the three that actually controls crawl access — path by path, bot by bot — and it's the only thing this tool evaluates for a pass/fail result. llms.txt is a different, much newer convention: a curated list of a site's key pages, meant to help a model navigate efficiently. It has no access-control effect whatsoever — publishing one doesn't block or allow anything, and no major AI vendor treats it as a gate.
Content-Signal is newer still and solves a third problem: once a bot has already fetched a page, what is the site owner's stated preference for how that content gets used — indexed for search, used to train a model, or used as live input to answer a question? It's declared per group, usually alongside the same Allow/Disallow lines, with values like ai-train=no or ai-input=yes. This checker surfaces it when present, but only as a reference next to the access result — it is a stated preference, not an enforcement mechanism, so it never changes whether this tool marks a path "Allowed" or "Blocked".
Allowed, blocked,
and the group that decided.
Each row reports one of two states. Allowed means the applicable group, evaluated under the rules above, has no rule that blocks the tested path — including the common case where there's no rule for that bot at all, which defaults to allowed. Blocked means a Disallow rule in that bot's applicable group is the longest match for the path. Next to each status, the "Matched rule" column names the exact group — the bot's own, or the fallback * — so you can see immediately whether a result comes from a rule written for that specific bot or from the general policy every unnamed crawler inherits.
That distinction is the one worth reading carefully. A result of "Blocked, matched: *" means you're relying on your general policy to cover a bot you never considered by name — which is fine until you add a new Disallow: / under * for an unrelated reason and quietly take every AI crawler down with it. A result of "Allowed, matched: GPTBot" means someone made that call for GPTBot specifically, on purpose.
Blocking *
and forgetting a bot.
The single most common error in a robots.txt is adding a blanket Disallow: / to the * group — to stop generic scrapers, or during a staging phase that was never cleaned up — without checking whether any named group already carves out an exception. If GPTBot has its own group with Allow: /, it stays allowed no matter what the * group says, because the more specific group always wins. But every AI crawler without a named group — which, on most sites, is most of them — silently inherits the new block the moment it ships.
The second version of the same mistake runs the other way: a site adds named Allow groups for the handful of AI crawlers it has heard of, assumes that covers "AI bots" as a category, and never notices that a dozen other named crawlers still fall through to a restrictive * group underneath. Pasting the real file into this tool and testing a handful of paths against the full bot list surfaces both versions in one pass — faster than reading the directives by eye and mentally resolving which group wins for each one.
About the
checker.
Go deeper on
crawler control.
Need the whole
policy reviewed?
AI crawler access as part of a free technical audit conversation — 24h response, no commitment.