Faragiri

Learn Robots and crawl control Technical

A leather bag maker blocked 4,000 filter addresses and was surprised to find forty of them still listed, with a URL and no description, four months later.

4Sections
4Min read
4Depth

A disallow stops the fetch, and a noindex removes the stored copy. Those are different jobs and the difference is the whole of this page. Bermondsey Bags makes leather bags in London, blocked about 4,000 filter addresses in February 2026 with one line in robots.txt, and four months later forty of them were still listed in results with a URL and no description.

Nothing had gone wrong. An address blocked in robots.txt can still be listed if something links to it, because the block prevents the fetch and the listing was built from the link rather than from the page. To remove a page you have to let it be fetched, so that the noindex on it can be read.

The file, and the three rules worth knowing

The file lives at the root and nowhere else. A robots.txt in a subdirectory is a text file nobody will ever read, and this catches people on sites where the shop sits in a folder.

File mechanics

Three robots.txt rules that trip up site owners

Crawlers only check the site root, match the longest pattern regardless of line order, and treat a blank Disallow as open access.

https://example.com/robots.txt
  1. 01/robots.txt (never /shop/robots.txt)
  2. 02Allow: /bags?filter=sale beats Disallow: /*?filter=
  3. 03Disallow:
  1. 01Must live at the rootAny robots.txt placed inside a subfolder is ignored completely.
  2. 02Longest matching rule winsPattern length overrides line order when Allow and Disallow overlap.
  3. 03Empty Disallow allows allA Disallow keyword with nothing after it blocks zero addresses.

Reading a robots.txt by eye often misses longest-match precedence and subfolder placement mistakes.

The longest matching rule wins, not the first one. A disallow on the filters and an allow on one filtered page below it works, and it works because of length rather than order, which is the opposite of how most people read a configuration file. Bermondsey's file has a disallow on the parameter and an allow on one address underneath it, and the allow wins because it is longer.

An empty Disallow line allows everything, which is the opposite of what it looks like. A file containing the word Disallow with nothing after it is a file that blocks nothing, and it appears in a surprising number of sites where somebody meant to write a rule and stopped.

The line that did the work

Disallow: /*?filter= blocks every address carrying that parameter, which at Bermondsey was about 4,000 combinations of colour, leather and size. The asterisk stands for anything, which is what makes one line enough, and the same line written without it matches nothing at all because no address begins with a question mark.

Two directives

What a disallow does, against what a noindex does

They are not two strengths of the same instruction, and using both on one address makes the weaker outcome permanent.

Disallow in robots.txtOne line, applies to thousands of addresses at once.
Noindex on the pageRead from the page, so the page has to be fetched.
Stops the fetchYesNo, it needs one
Removes a stored copyNoYes
Stops the address being listedNo, a link is enoughYes
Costs crawl every timeNoYes, one fetch per page
Right for 4,000 filter addressesYes, one lineNo, 4,000 fetches

The last two rows are why both exist. Neither is better; a disallow is for volume nobody needs to reach and a noindex is for a page a person still uses.

What it bought was crawl. The shop has 90 products and a crawler was fetching thousands of pages of the same 90, which is how a page is found in the first place being spent on addresses nobody wanted. What it did not buy was removal, and the forty listed addresses had to be handled the other way round: the disallow came off, each returned a noindex, they left over about five weeks, and the disallow went back on afterwards.

Where this stalls: two locks on the same door

The stall is doing both at once, and it is done by careful people on the reasonable theory that two instructions are safer than one. A disallow and a noindex on the same address cancel: the noindex is inside a page that will never be fetched, so nothing ever reads it, and the address stays listed indefinitely.

Correct order

How to remove already-listed URLs before blocking crawl

Putting disallow and noindex on together locks the URL in search; you have to allow the fetch first so noindex can be read.

  1. OpenLift the disallowLet crawlers fetch the listed filter URLs again.
  2. ReadServe noindexEach fetched page delivers its removal directive.
  3. ClearWait for dropListed addresses fall out of search over roughly five weeks.
  4. BlockRestore disallowRe-apply the crawl block only if crawl waste requires it.

Two locks on the same door cancel each other out because a blocked crawler never sees the noindex tag inside the page.

The order that works is the opposite of the intuitive one. Allow the fetch, let the noindex be read, wait for the addresses to fall out, and only then block the crawl if there is a crawl reason to. Nobody arrives at that order by reasoning about it, which is why it is worth writing down.

Two smaller facts

Crawl-delay is ignored by Google and honoured by some others. It is in a great deal of old advice and it is not a way to control Google's crawl rate; the setting for that lives in Search Console and is rarely the right answer at any size where somebody is reading this.

Live verification

Checking what crawlers actually receive from robots.txt

Do not guess from a local text editor or rely on Crawl-delay; test live responses and specific URL matches.

Robots.txt checks

  1. YesTest URLs in Search Console reportShows the exact fetched file and which longest rule matched.Yes
  2. YesInspect live headers with curlCatches silent server faults like Bermondsey's three-week 404 in 2025.Yes
  3. NoUse Crawl-delay for GoogleGoogle ignores Crawl-delay even though some other bots honour it.No

Browsers follow redirects and hide raw status codes, whereas curl prints the exact HTTP response a crawler sees.

Test the file against a specific address rather than reading it. Reading a robots.txt and reasoning about it is how the longest-match rule gets forgotten. Search Console's robots.txt report shows the fetched file and which rule applies to an address you type, free, and it replaced pasting the file into a text editor and squinting at asterisks.

curl fetches an address the way a crawler does and prints the status and headers, free and already on the machine, and it replaces trusting a browser, which follows redirects silently and runs script before showing anybody anything. It is the only way to see what a robots.txt actually returns rather than what the server was supposed to return, and at Bermondsey it returned a 404 for three weeks in 2025 without anybody noticing.

Which addresses should be blocked rather than merged is naming the real version of a duplicated page, what is deliberately listed instead is listing what you want indexed, and all of it belongs to the layer that decides whether any of the rest counts.

Alongside this

Above this