Pikkola Crawl Studio, tool No. 38

The file a crawler reads first.

Choose the rules instead of typing them, paste a file you already have to have its mistakes named by line, test a path the way a crawler matches it, and write the sitemap that points at the pages you want found. Nothing leaves this tab.

2
files written
0
requests sent
3
languages

The file

Write it by choosing, not typing

Each group below becomes a block of text. What you see on the right is the file you publish, with its line numbers.

Start from

Only used to write the Sitemap line in a preset.

Sitemaps

A file you already have

Paste it, and read it line by line

A plugin wrote it, or a colleague did. Paste it here to see how a crawler reads it: the groups it finds, the lines that do nothing, and the ones that do something other than intended.

    A note is not always a mistake. It is what a crawler will do with the line.

    The test

    Ask the crawler about one path

    This applies the rule the way it is published: the group for that crawler first, then the longest matching pattern, with Allow winning a tie. The line that decided it is marked in the file.

    A full address works too. Only the path and the query are matched.

    Allowed

    Rules that matched

      Two rules of the same length: Allow wins.

      The map

      A sitemap from a list of addresses

      Paste the pages you want found, one per line. A date, a change frequency and a priority apply to all of them, and any row can be changed on its own.

      For every address

      One sitemap holds up to 50,000 addresses and 50 MB. Past that, split the file and point at the parts with a sitemap index.

      Each address

      sitemap.xml
      
                

      How it is read

      Three things decide what a crawler does

      01

      The group

      A crawler looks for the group whose User-agent token appears in its own name, the longest token first, and falls back to *. Two groups that name the same crawler are read as one.

      02

      The longest path

      Every Allow and Disallow whose pattern matches is collected and the longest pattern decides. A pattern is a prefix unless it ends with $, and * stands for any run of characters.

      03

      The tie

      When an Allow and a Disallow match at the same length, Allow wins. Nothing else in the file, Crawl-delay and Sitemap included, decides access.

      Who reads which line
      Line Google Bing Others
      Allow, Disallowreadsreadsreads
      Crawl-delayignoresreadssome
      Sitemapextension, widely readextension, widely readsome
      Noindexnot a directivenot a directivenot a directive

      Questions

      Do I need a robots.txt at all?

      No. Without the file every crawler may read everything. Add one when something should be left out, and keep the Sitemap line in it so the map is found without being submitted by hand.

      Does Disallow keep a page out of search results?

      Not by itself. A blocked page can still be listed if other pages link to it, with no snippet. To keep a page out of the index, let it be crawled and answer with noindex, or send the X-Robots-Tag header.

      Will blocking GPTBot keep my text out of AI answers?

      Only out of the crawlers that obey the file. GPTBot, ClaudeBot and CCBot read robots.txt. A crawler that ignores it, or a person who copies the page, does not. The file is a request, written down, not a lock.

      Why is my Sitemap line ignored?

      Two usual reasons: it is a path rather than a full https:// address, or it sits inside a group where the crawler does not expect it. Write it as a full address, in its own block at the end.

      Should a Next.js site block /_next/?

      No. Google renders the page with its scripts, and a blocked script leaves an empty page in the index. Allow /_next/static/, and block your own /api/ and /admin/ instead.

      Is anything I type sent anywhere?

      No. The file is held in this tab and kept in this browser's local storage so a reload does not lose your work. Publishing it is a copy, a paste or a download on your side.

      Why not put every rule in the * group?

      Because one crawler often needs an exception. A group that names Googlebot is read instead of the * group, not in addition to it, so the exception has to be complete.