01
The group
A crawler looks for the group whose User-agent token appears in its own name, the longest token first, and falls back to *. Two groups that name the same crawler are read as one.
Pikkola Crawl Studio, tool No. 38
Choose the rules instead of typing them, paste a file you already have to have its mistakes named by line, test a path the way a crawler matches it, and write the sitemap that points at the pages you want found. Nothing leaves this tab.
The file
Each group below becomes a block of text. What you see on the right is the file you publish, with its line numbers.
Only used to write the Sitemap line in a preset.
No groups yet. Add one, or start from a preset above.
A file you already have
A plugin wrote it, or a colleague did. Paste it here to see how a crawler reads it: the groups it finds, the lines that do nothing, and the ones that do something other than intended.
A note is not always a mistake. It is what a crawler will do with the line.
The test
This applies the rule the way it is published: the group for that crawler first, then the longest matching pattern, with Allow winning a tie. The line that decided it is marked in the file.
A full address works too. Only the path and the query are matched.
Two rules of the same length: Allow wins.
The map
Paste the pages you want found, one per line. A date, a change frequency and a priority apply to all of them, and any row can be changed on its own.
One sitemap holds up to 50,000 addresses and 50 MB. Past that, split the file and point at the parts with a sitemap index.
How it is read
01
A crawler looks for the group whose User-agent token appears in its own name, the longest token first, and falls back to *. Two groups that name the same crawler are read as one.
02
Every Allow and Disallow whose pattern matches is collected and the longest pattern decides. A pattern is a prefix unless it ends with $, and * stands for any run of characters.
03
When an Allow and a Disallow match at the same length, Allow wins. Nothing else in the file, Crawl-delay and Sitemap included, decides access.
| Line | Bing | Others | |
|---|---|---|---|
| Allow, Disallow | reads | reads | reads |
| Crawl-delay | ignores | reads | some |
| Sitemap | extension, widely read | extension, widely read | some |
| Noindex | not a directive | not a directive | not a directive |
No. Without the file every crawler may read everything. Add one when something should be left out, and keep the Sitemap line in it so the map is found without being submitted by hand.
Not by itself. A blocked page can still be listed if other pages link to it, with no snippet. To keep a page out of the index, let it be crawled and answer with noindex, or send the X-Robots-Tag header.
Only out of the crawlers that obey the file. GPTBot, ClaudeBot and CCBot read robots.txt. A crawler that ignores it, or a person who copies the page, does not. The file is a request, written down, not a lock.
Two usual reasons: it is a path rather than a full https:// address, or it sits inside a group where the crawler does not expect it. Write it as a full address, in its own block at the end.
No. Google renders the page with its scripts, and a blocked script leaves an empty page in the index. Allow /_next/static/, and block your own /api/ and /admin/ instead.
No. The file is held in this tab and kept in this browser's local storage so a reload does not lose your work. Publishing it is a copy, a paste or a download on your side.
Because one crawler often needs an exception. A group that names Googlebot is read instead of the * group, not in addition to it, so the exception has to be complete.