The rule
Next.js serves a payload alongside each page for client-side navigation. Those live at addresses like /contact/index.txt, and they are of no use to a search engine. So the file said:
Disallow: /index.txt$
Disallow: /*/index.txt$Read it aloud and it is obviously right. Every index.txt, at the root and in any directory, blocked. It was in production for weeks.
Why it matched nothing
$ anchors the pattern to the end of the URL — not the end of the path. And Next does not request those payloads bare. It appends a build stamp:
/contact/index.txt?_rsc=gzsk8That URL does not end in .txt. It ends in gzsk8. The anchor failed, the rule did not apply, and Googlebot fetched every one of them. Crawl stats put the cost at 35% of every request Google made to the site in a week — budget spent on files that exist only to make a link click feel fast.
The part that makes it dangerous
Nothing reported it, and nothing could have. A directive that matches nothing produces exactly the same output as a directive with nothing to match: a line of text and no effect. There is no error, no warning, no count that goes to zero. The file is valid. The only symptom is in a crawl report nobody reads weekly.
Removing the two $ characters fixed it. The prefix form matches anything beginning with that path, query string included, which is what was meant all along.
The three rules, in full
Google's matcher has three behaviours and no others. * stands for any run of characters. $ anchors to the end of the URL. Everything else is a literal prefix — so Disallow: /admin blocks /administrator-guide/ as well, which catches people out in the opposite direction.
Precedence is by pattern length, not by position. The longest matching rule wins wherever it is written, and a tie goes to Allow. That is what lets Allow: /public/reports/ rescue a subtree from Disallow: /public/ even when the Allow is written first, or last, or between two unrelated lines.
Two things robots.txt does not do
It does not remove a page from search. Blocking stops the fetch, not the listing. Google can and does list a URL it was told not to crawl, with no snippet underneath, on the strength of links pointing at it. To keep a page out of the index, let it be crawled and give it a noindex tag — the two instructions are mutually exclusive, and blocking wins by preventing the tag from ever being read.
It does not hide anything. The file is public and often the first thing an attacker reads. Listing /internal-admin/ there advertises it. Use authentication for anything that matters.
Testing it properly
Test with the query string, not without. That is the shape crawlers actually request and the shape an anchored rule misses. Check both directions: what must be blocked, and what must stay reachable — /_next/, /assets/, ads.txt, the sitemap.
The robots.txt tester evaluates a URL with the matcher above and names the rule that decided it, which is the half of the answer that tells you what to change. Point the finished file at a sitemap built by the sitemap generator.