About
Last updated 20 November 2026
A robots.txt is one of the last files on a site that is written for a machine rather than a person, and it is usually pasted from something found in a hurry. This page treats it as text with rules: written by choosing, read line by line, and tested the way a crawler tests it.
What it does
- The builder. A group is a list of crawler names and a list of Allow, Disallow and Crawl-delay lines. Sitemaps sit in their own block. The file appears beside it with line numbers, and leaves as a copy or a download.
- The reader. Paste a file and every line is parsed: the groups it finds, the line numbers of the rules inside them, and the lines that do nothing or do the opposite of what was intended. An empty Disallow, a Sitemap written as a path, a Noindex line and a full address used as a path are all named where they sit.
- The test. One user-agent and one path. The page finds the group a crawler would use, collects every Allow and Disallow that matches, and shows the winner with the line number it came from. The losing matches stay on screen, because the reason a rule did not win is usually the thing that was misunderstood.
- The map. A list of addresses becomes sitemap.xml with lastmod, changefreq and priority, set once for all of them or one row at a time. The same list can be saved as plain text.
How the matching is done
It follows the published behaviour rather than a guess. A crawler picks the group whose User-agent token is contained in its own name, and the longest token beats a shorter one, with * as the fallback. Groups that name the same crawler are merged, as Google does. Within those rules, every pattern that matches is collected, the longest pattern decides, and Allow wins a tie.
A pattern is a prefix match unless it ends with $, and * stands for any run of characters. /admin/ therefore matches /admin/login but not /administration. The length compared is the length of the pattern as written, which is how the rule is stated in the specifications.
Where the rules come from
The parsing and matching follow RFC 9309, the Robots Exclusion Protocol, and the crawler documentation that Google and Bing publish beside it. Crawl-delay is not in the standard: it is a convention that Bing and Yandex still read and Google ignores, so the page says so instead of pretending otherwise. Sitemap is an extension of the same file, widely read, and is why it gets its own block here.
What it deliberately does not do
- It does not fetch your site or check whether the file is live. There is no request leaving the page, so it cannot know what your server currently answers.
- It does not tell you whether a page is indexed, or predict how a search engine will rank it. A robots.txt decides what may be read, not what is shown.
- It does not publish the file. The publish step is a copy into your repository or a download onto your server.
- It does not score your AI visibility or send your file anywhere for analysis. The content policy of the page is
connect-src 'none', so there is nothing that could receive it.
Why it is free
The whole studio is a static page: no server, no account, no database, nothing to meter. It sits with the other Pikkola tools, each of which does one small job in the browser. There is no advertising or tracking on the page, and no plan to add a paid tier to this one.
Contact
Write to hello@pikkola.com. A wrong verdict with the file and the path that produced it is the most useful message you can send.