Robots.txt Generator and Tester
Build a robots.txt and test any URL against it using the same matching rules Google uses, including the anchor that silently makes a rule match nothing.
Processed on your device — no upload
| URL | Result | Why |
|---|---|---|
| / | Allowed | Allow: / — The only matching rule. |
| /admin/users | Blocked | Disallow: /admin/ — 2 rules match; the longest pattern wins, and a tie goes to Allow. |
| /products/?sort=price | Blocked | Disallow: /*?sort= — 2 rules match; the longest pattern wins, and a tie goes to Allow. |
| /blog/post-1/ | Allowed | Allow: / — The only matching rule. |
Matching uses Google’s three rules and only those: * stands for any run of characters, $ anchors to the end of the URL, and everything else is a literal prefix. The longest matching pattern wins wherever it is written in the file, and a tie goes to Allow.
Test a URL with its query string, not without. A rule ending in $ stops matching the moment anything follows — which is how a directive that reads correctly can block nothing at all, silently, for as long as nobody checks.
Blocking a page in robots.txt does not remove it from search. Google can still list a URL it was told not to fetch, with no description under it. To keep a page out of the index, let it be crawled and give it a noindex tag.
How to use it
- Write or paste your rules.
- List the URLs you care about.
- Read which rule decided each one.
How it works
Matching uses Google's three rules and only those. * stands for any run of characters, $ anchors the pattern to the end of the URL, and everything else is a literal prefix. Precedence is by pattern length rather than by position in the file, so the most specific rule wins wherever it happens to be written, and a tie goes to Allow.
The generator half is the easy half. A robots.txt cannot be read for correctness: a rule that matches nothing looks exactly like a rule with nothing to match. Both are a line of text with no effect, and only one of them is a bug.
This site shipped that bug. Disallow: /*/index.txt$ sat in production for weeks. It parses, it validates, and it blocked nothing at all — $ anchors to the end of the URL, and the payloads were requested with a build stamp appended, so /contact/index.txt?_rsc=gzsk8 never ended in .txt. Thirty-five percent of a week's crawl budget went to pages that should never have been fetched. Nothing reported it, because there was nothing to report.
So test with the query string, not without. That is the shape the anchor misses, and it is the shape crawlers actually request.
A worked example
The test table names the deciding rule, not just the verdict. When four rules match one URL, knowing that it is blocked is half an answer; knowing which line blocked it is the whole one. The sample includes an Allow rescuing a subtree from a broader Disallow, which is the case where precedence stops being obvious.
Check what must stay reachable. The commonest damage from a robots.txt is not a page left crawlable — it is /_next/, /assets/ or /static/ blocked wholesale, which hides the CSS and JavaScript Google renders the page with. The result is a site that scores badly for reasons nobody can see in the HTML.
Blocking is not removing. A URL disallowed here can still be listed in search, with no description under it, because Google saw it linked from elsewhere and was told not to look. To keep a page out of the index, let it be crawled and give it a noindex tag — the two instructions are mutually exclusive, and blocking wins by preventing the tag from ever being read.
Once the file is right, point it at a sitemap built by the sitemap generator.
Questions
Where does robots.txt go?
At the root of the host, and nowhere else — https://example.com/robots.txt. A file in a subdirectory is ignored, and a subdomain needs its own.
What does the $ do?
It anchors the pattern to the end of the URL. It is the single most common source of rules that silently match nothing, because any query string or fragment after the path makes the anchor fail.
Which wins when two rules match?
The longer pattern, wherever it sits in the file. If they are the same length, Allow wins. Order in the file has no effect at all.
Does Disallow keep a page out of Google?
No. It stops the fetch, not the listing. A blocked URL can still appear with no snippet. Use a noindex tag and allow the crawl, or the tag is never read.
Does Google support crawl-delay?
No, and it never has. Bing does. Google’s crawl rate is adjusted in Search Console, not in this file.
Can I put noindex in robots.txt?
It was an undocumented behaviour that was removed in 2019 and never supported before that. It does nothing now, so this reports it rather than emitting it.
Should I block my admin area?
Only if the URLs are not secret. robots.txt is public, so listing a path there advertises it. Anything sensitive needs authentication, not a polite request.
Is Robots.txt Generator and Tester free to use online?
Yes — free, with no account, no daily limit and no watermark. It runs online in your browser, and because the work happens on your own device it keeps working offline once the page has loaded.
Related SEO tools
XML Sitemap Generator
With the language alternates
open$termiva hreflang --checkHreflang Tag Generator
Clusters that Google keeps
open$termiva meta --analyseMeta Tag Analyser
Paste the source, read the verdict
open$termiva density --phrases 3Keyword Density Checker
Counts Arabic properly
open