Enter a website and a page. The checker reads its robots.txt the way Google does and shows whether Googlebot, Bingbot and the main AI crawlers may crawl that page, and which rule decides.
robots.txt is a plain text file at the root of a website (/robots.txt) that tells crawlers which addresses they may fetch. Well-behaved crawlers like Googlebot and Bingbot follow it.
It controls crawling, not indexing. A page blocked in robots.txt can still appear in Google if other sites link to it, just without a description. To keep a page out of search results, let it be crawled and add a noindex robots meta tag instead.
User-agent: Googlebot), or the User-agent: * group if none does.* matches any characters and $ marks the end of the address, so Disallow: /*.pdf$ blocks PDF files only.Disallow: allows everything; Disallow: / blocks the whole site.There are two kinds. Training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended, meta-externalagent) collect pages to train models; blocking them doesn't change your Google or Bing rankings. Search and assistant crawlers (OAI-SearchBot, ChatGPT-User, Claude-User, PerplexityBot) fetch pages to answer questions and can send visitors with a link back; blocking them keeps your pages out of those answers.
A common choice is to allow search-type crawlers and decide about training crawlers case by case. Either way, check the result here after you change the file.
The Serpzeno SEO audit checks robots.txt, your XML sitemap and whether pages are blocked from Google by robots rules or noindex, on every page, every week.
Always at the root of the domain, for example https://example.com/robots.txt. Each subdomain (shop.example.com) and each protocol has its own file.
No. It stops crawling, not indexing. Use a noindex robots meta tag (and leave the page crawlable) to remove a page from search results.
No. Training crawlers aren’t used for Google or Bing rankings. Blocking search-type AI crawlers like OAI-SearchBot or PerplexityBot does keep you out of their answers.
Google reads the first 500 KB and ignores the rest. Most sites need only a few lines.
Yes, with no sign-up. It fetches only the public robots.txt file of the site you enter and doesn’t store it.
This tool checks one thing at a time. The Serpzeno audit runs 28 checks, from titles and descriptions to broken links and page speed, on every page of your site, every week, on the Free plan.