About

Last updated 20 November 2026

A robots.txt is one of the last files on a site that is written for a machine rather than a person, and it is usually pasted from something found in a hurry. This page treats it as text with rules: written by choosing, read line by line, and tested the way a crawler tests it.

What it does

How the matching is done

It follows the published behaviour rather than a guess. A crawler picks the group whose User-agent token is contained in its own name, and the longest token beats a shorter one, with * as the fallback. Groups that name the same crawler are merged, as Google does. Within those rules, every pattern that matches is collected, the longest pattern decides, and Allow wins a tie.

A pattern is a prefix match unless it ends with $, and * stands for any run of characters. /admin/ therefore matches /admin/login but not /administration. The length compared is the length of the pattern as written, which is how the rule is stated in the specifications.

Where the rules come from

The parsing and matching follow RFC 9309, the Robots Exclusion Protocol, and the crawler documentation that Google and Bing publish beside it. Crawl-delay is not in the standard: it is a convention that Bing and Yandex still read and Google ignores, so the page says so instead of pretending otherwise. Sitemap is an extension of the same file, widely read, and is why it gets its own block here.

What it deliberately does not do

Why it is free

The whole studio is a static page: no server, no account, no database, nothing to meter. It sits with the other Pikkola tools, each of which does one small job in the browser. There is no advertising or tracking on the page, and no plan to add a paid tier to this one.

Contact

Write to hello@pikkola.com. A wrong verdict with the file and the path that produced it is the most useful message you can send.

Back to the studio