Faragiri

Learn Technical SEO · Robots and crawl control, Sitemaps, Canonical tags and 4 more

Sixty-one products and 9,400 crawlable addresses in Aberdeen, and the one question that decides which of the four tools each address needs.

7Branches
7Nodes below
3Depth

Braemore Instruments in Aberdeen sells refurbished laboratory equipment: 61 products, and a crawl in January 2026 found 9,400 addresses. Four filters on the catalogue, sort order, manufacturer, condition and page number, combine with each other, and every combination is a working address returning a page. 9,400 addresses from 61 products, because four filters combine.

None of those addresses is wrong exactly. Each one loads, each one shows equipment, and a person clicking through the shop would never see a problem. What they cost is the crawl, and what they create is 9,400 candidates for a question with 61 answers.

Four directives, and what each one does not do

The confusion that causes most of the damage here is that the four available instructions sound interchangeable and behave nothing alike.

Four directives, plus hreflang

What each instruction stops, and what carries on regardless

Every directive has a second half, and misunderstanding that second half is why people combine rules that cancel each other out.

  1. robots.txt disallowThe fetch, for any address matching the patternAnything already listed, which stays listed with no description
  2. noindexThe stored copy, once the page has been fetchedThe crawl itself, which carries on costing a fetch each time
  3. CanonicalTwo addresses being counted as two pages, usuallyAny case the engine disagrees with, silently, because it is a hint
  4. 301 redirectThe old address resolving at all (unlike temporary 302)Almost everything the old address earned, which is the point of it
  5. hreflangThe wrong language showing in the wrong countryEvery tag the other page does not return, which is ignored whole
The index

A temporary 302 left in place for a year tells engines the old address is coming back, leaving a move unfinished.

A robots.txt disallow stops the fetch. It does not remove anything already stored, and an address that was listed before the rule went in stays listed. A noindex removes the stored copy but has to be fetched to be obeyed, so it costs a crawl every time.

A canonical is a hint and can be ignored; a redirect is not a hint. That distinction is the one worth carrying: pointing a canonical from a filtered address at the plain product page is a request that is usually granted and sometimes declined, and when it is declined nothing tells you. A 301 removes the choice, because the old address stops resolving.

The version numbers matter too. A 302 says the old address is coming back, and it will be treated that way, which is why a temporary redirect left in place for a year is a page that never finished moving.

The question that picks the right one

The decision is not four-way. It is one question: does anything outside the site link to this address? If something does, the address has value and the job is to move that value, with a 301 for a page that has moved or a canonical for a duplicate that has to stay reachable. If nothing does, the address has no value and the job is to stop it cheaply, with a noindex if a person still needs to reach it and a disallow if nobody does and there are thousands.

Choosing between them

One question decides which of the four an address needs

Not a four-way lookup. The tools divide on whether the address has anything worth carrying, and everything else follows from that.

Does anything outside the site link to this address?
  • Something does
    Move the valueThe address has earned something, so it has to arrive somewhere rather than stop.
    • 301 for a page that has moved for good
    • Canonical for a duplicate that has to stay reachable
  • Nothing does
    Stop it cheaplyThere is nothing to carry, so the only question left is what it costs to hold shut.
    • noindex where a person still needs to reach it
    • Disallow where nobody does and there are thousands

At Braemore the question sorted 9,400 addresses into 75 with links (61 products, 14 categories) and the rest without.

Braemore's 9,400 split on exactly that line. 61 product pages and 14 category pages had links from outside. The remaining addresses had none, and a disallow on the filter parameters took the crawl from 9,400 to 75 without any page a person could reach disappearing.

Redirects point at the destination, not at the next redirect

Two site changes in four years had left chains. An old product address went to a category, which went to a renamed category, which went to the new catalogue. Every hop costs a fetch and the crawler stops following after a few of them, so the last address in the chain is reached late or not at all.

Direct hops

Two site changes left multi-hop chains that exhaust crawler patience

Point every old address straight at the final destination rather than stacking new redirect lines on top of old ones.

Adding one new rule takes seconds, whereas rewriting old rules requires reading the existing redirect table.

Point a redirect at the final address, not at another redirect. That means rewriting the old rules when a new one is added, and the reason it does not happen is that adding a rule is one line and rewriting the old ones means reading them.

Two of the tags have a rule nobody reads

Braemore sells into Ireland from a second set of pages. A hreflang tag that is not returned by the other page is ignored, and the Irish pages pointed at the British ones while the British pages pointed at nothing. The whole arrangement did nothing for eleven months, and it also did no harm, which is the worst possible outcome because nothing ever reported it.

Silent tag failures

Two directives that fail quietly without raising a single error

Unreturned hreflang tags and disallowed sitemap URLs contradict themselves silently, wasting months with no alert.

One-way hreflangIrish pages pointed to British pages; British pointed nowhere
Disallowed sitemap URLListed in XML sitemap while blocked in robots.txt
What was configuredRegional annotation on only one side of the pairTwo conflicting files giving opposite instructions
How the engine reactsIgnores the unreciprocated tag completelyTreats sitemap as a suggestion and obeys the block
Why it persistsDid nothing for eleven months with no harm reportedTeams assume the more optimistic file will win

A sitemap only declares what exists; it never overrides a disallow rule or forces indexing.

A sitemap has the mirror-image problem. It is a suggestion about what exists, not a request to index, and a page in a sitemap that is disallowed in robots.txt is a site giving two instructions and expecting the more optimistic one to win.

Where this stalls: the layer with nobody to sell it

The stall is ownership. This layer belongs to whoever deploys the site, and the person who notices the problem is usually somebody else with no access. So the finding gets written into a document, the document goes into a backlog behind features, and the 9,400 addresses stay because nobody's quarter is worse for them.

Ownership gap

Why technical fixes sit in the backlog while content plans ship on broken ground

The person spotting crawl waste lacks deploy access, leaving 9,400 filter URLs and stale canonicals untouched behind product features.

  1. Spotted in audit2
    • Crawl waste9,400 filter URLs
    • Canonical to 2024 deleted page
  2. Backlog stall2
    • Ownership gapNo deploy access for auditor
    • Nobody's quarterQueued behind feature work
  3. Active work1
    • Targets broken categoryFresh content plan

A two-year-old technical audit paired with a fresh content plan is the clearest sign of split ownership.

The tell is a site with a two-year-old audit and a fresh content plan. Braemore had both, and the content plan was for a category that had been carrying a canonical pointing at a page deleted in 2024.

Two things to check with in 2026

curl asks for a page the way a crawler does and prints the status code and the headers, which is the only way to see a redirect chain as a chain. It is free, it is already on the machine, and it replaces trusting a browser, which follows every hop silently and runs the script before showing anybody anything.

Two checks in 2026

Use curl for raw headers, Removals for emergencies, and GSC one URL at a time

Browsers hide redirect hops and run scripts first; curl prints the raw crawler response while Removals buys six months to add noindex.

  • curl commandAlready on the machine, free
    • Prints raw HTTP status codes and headers
    • Exposes every hop in a redirect chain
  • Canonical inspectionNo bulk shortcut
    • Search Console reports chosen URL one at a time
    • Impractical to inspect 9,400 addresses manually
2026 technical checks
  • GSC Removals toolHides in a day for six months
    • Emergency cover for confidential pages
    • Temporary only; underlying noindex is still required

Neither curl nor the Removals tool can tell you globally whether your canonical hints were honoured.

The Removals tool in Google Search Console hides an address from results in about a day, for roughly six months. It is free, and it replaces waiting for the next crawl when something confidential has gone up. It is worth knowing that it hides rather than fixes: the six months run out, and if the underlying noindex is not in place the address returns.

Neither one knows whether a canonical was honoured. Search Console reports the address it chose, one at a time, and for 9,400 addresses there was never going to be an afternoon long enough.

The seven pieces of the layer

The disallow that fixed the crawl is what to let bots see, and what to keep out, and the file listing the 61 is listing what you want indexed. The hint that can be declined is naming the real version of a duplicated page, and the chain is moving a page without losing what it earned.

The seven pieces

Seven controls built on fetch vs store and what a site links together

Dropping Braemore's crawl from 9,400 to 75 showed nothing measurable on 61 products, though a 61,000-product shop would feel the gain immediately.

Fetch and store
  • robots.txt disallowBlocks crawler fetches on parameterised waste
  • XML sitemapsLists the 61 real product pages you want indexed
  • Canonical hintsNames the preferred URL among duplicate views
  • 301 redirectsMoves earned equity directly without chains
  • Pre-script HTMLWhat crawlers inspect before JavaScript executes
  • Migration checklistPrevents traffic drops during deadline launches
  • Reciprocal hreflangServes Irish and British pages without self-competition

Why 9,400 addresses happened is structural as well as technical, grounded in how fetch and store behave separately.

What the catalogue looks like before the script runs is what a crawler sees when the page needs script. Doing all of it at once, on a deadline, is the checklist, and the traffic drops it prevents, and the Irish pages are serving the right language without competing with yourself.

Why a fetch and a store are separate events, which is what makes half of this behave the way it does, is the mechanism, before any tactic makes sense. Why 9,400 addresses is a structural fact and not only a technical one is how pages relate, which decides what a site is about.

The crawl went from 9,400 to 75 in February 2026 and nothing measurable happened for four months. Whether the saved crawl was ever the constraint at a site with 61 products is a question Braemore cannot answer, and a shop with 61,000 would not have had to ask.

Inside this, 7

Alongside this

Above this