Robots.txt checker for Google and AI crawlers

Enter a website and a page. The checker reads its robots.txt the way Google does and shows whether Googlebot, Bingbot and the main AI crawlers may crawl that page, and which rule decides.

What robots.txt does, and what it doesn’t

robots.txt is a plain text file at the root of a website (/robots.txt) that tells crawlers which addresses they may fetch. Well-behaved crawlers like Googlebot and Bingbot follow it.

It controls crawling, not indexing. A page blocked in robots.txt can still appear in Google if other sites link to it, just without a description. To keep a page out of search results, let it be crawled and add a noindex robots meta tag instead.

How Google reads the rules

  • A crawler follows only one group: the one naming it (for example User-agent: Googlebot), or the User-agent: * group if none does.
  • Within that group, the longest matching path wins. When an Allow and a Disallow are the same length, Allow wins.
  • * matches any characters and $ marks the end of the address, so Disallow: /*.pdf$ blocks PDF files only.
  • An empty Disallow: allows everything; Disallow: / blocks the whole site.
  • A missing robots.txt (404) means no restrictions. A server error (5xx) makes Google pause crawling the whole site, so a broken robots.txt is worse than none.

Should you block AI crawlers?

There are two kinds. Training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended, meta-externalagent) collect pages to train models; blocking them doesn't change your Google or Bing rankings. Search and assistant crawlers (OAI-SearchBot, ChatGPT-User, Claude-User, PerplexityBot) fetch pages to answer questions and can send visitors with a link back; blocking them keeps your pages out of those answers.

A common choice is to allow search-type crawlers and decide about training crawlers case by case. Either way, check the result here after you change the file.

Check robots.txt as part of a full audit

The Serpzeno SEO audit checks robots.txt, your XML sitemap and whether pages are blocked from Google by robots rules or noindex, on every page, every week.

Questions

Where is my robots.txt file?

Always at the root of the domain, for example https://example.com/robots.txt. Each subdomain (shop.example.com) and each protocol has its own file.

Does blocking a page in robots.txt remove it from Google?

No. It stops crawling, not indexing. Use a noindex robots meta tag (and leave the page crawlable) to remove a page from search results.

Does blocking GPTBot or ClaudeBot hurt my SEO?

No. Training crawlers aren’t used for Google or Bing rankings. Blocking search-type AI crawlers like OAI-SearchBot or PerplexityBot does keep you out of their answers.

How big can robots.txt be?

Google reads the first 500 KB and ignores the rest. Most sites need only a few lines.

Is this checker free?

Yes, with no sign-up. It fetches only the public robots.txt file of the site you enter and doesn’t store it.

Check every page at once

This tool checks one thing at a time. The Serpzeno audit runs 28 checks, from titles and descriptions to broken links and page speed, on every page of your site, every week, on the Free plan.

Audit my whole site free