Faragiri

Learn Indexing How search works

A Lincolnshire tile shop had 2,880 addresses, 2,410 of them fetched and 1,150 kept, and the ones that fell out were near copies of each other.

4Sections
4Min read
4Depth

Being fetched decides nothing. Thurlby Tiles, which sells floor and wall tiles from a yard in Bourne, had 2,880 addresses known, 2,410 fetched, 1,150 stored, 780 ever shown in the twelve months to July 2026, and the interesting gap is not where anybody expects it.

The biggest drop is between fetched and stored, not between known and fetched. 1,260 pages were read in full by a crawler and then not kept, which is a judgement rather than a failure: something read them, decided they added nothing to what was already stored, and moved on.

Why 1,260 pages were not worth keeping

Nearly all of them were size variants. A tile sold in four sizes had four addresses, each with the same photograph, the same description and one number different. Near duplicate pages are dropped as a set, keeping one, and which one is kept is not a decision the shop made.

Indexing drop

Addresses known against pages actually stored and shown

The sharpest drop sits between fetched and stored, where 1,260 pages were read in full and discarded as near-duplicate size variants.

  1. Addresses knownFrom internal links and the sitemap over twelve months to July 2026.2,880
  2. FetchedRead in full by a crawler.2,410
  3. Stored1,260 pages dropped, mostly duplicate tile size URLs.1,150
  4. Ever shown for anythingMatched to at least one query.780

Merging 640 tile size variants into 90 pages placed 74 of them into results within six weeks.

An index stores one processed copy per distinct thing it believes it has. A page that differs only in a size or a colour is one page to an index, and arguing with that is arguing with the storage model rather than with a ranking. Thurlby merged the variants: merging 640 size variants into 90 pages put 74 of them into results within about six weeks, with the sizes as options on one page.

The canonical you declared and the one that was chosen

The shop had declared a canonical on every variant, pointing at the largest size, which is exactly the right instinct. It made no difference to about a fifth of them, because the engine picks the canonical and your declaration is a suggestion.

Canonical signals

The URL you declared versus the URL the engine stored

On 210 pages, Thurlby declared one canonical while Search Console reported another selected because internal links outweighed the tag.

  • Site-declared canonicalTreated as a suggestion
    • Pointed every variant at the largest tile size
    • Ignored on about a fifth of variant URLs
    • Weak when the target URL has no internal links
  • Engine-selected canonicalActual URL in the index
    • Listed separately in Search Console page indexing
    • Favours the URL linked from thirty places
    • Follows link weight over an unlinked tag

210 pages with declared vs chosen mismatch

Always check the engine-selected URL in Search Console rather than assuming your canonical tag won.

The way to see this is in Search Console's page indexing report, where the two are listed separately: the URL declared by the site and the URL selected by the engine. Check which URL the engine chose, not which one you declared. Thurlby had 210 pages where the two disagreed and had been reading the declaration as though it were the outcome for a year.

Where they disagree, the reason is usually that the chosen page has links pointing at it and the declared one does not. A canonical is one signal among several rather than an instruction, and a page linked from thirty places will win against a tag on a page linked from none.

Where this stalls: writing more of what was dropped

The stall is producing more pages after a drop. Thin and duplicate are the two reasons that account for most drops, and both of them get worse with volume: a shop that responds to 1,260 dropped variants by generating another 400 has added more of exactly what was refused.

Storage gate

Why publishing more thin or duplicate pages only increases drops

Adding 400 more pages after 1,260 drops feeds the exact condition that triggers refusal: nothing distinct to store.

  1. Duplicate size sets1,260 dropped variants and 400 new copiesOne merged product page with size options
  2. Thin supplier blurbsForty repeated words, photo, and priceContent no stored page already contains
  3. Catalogue-wide cleanupScaling identical templates across rangesMerging variants across the full catalogue
What passes vs what gets refused

Refusal at storage is a deduplication decision, not a quality penalty.

Doing the merge across a whole catalogue rather than one range is search for sites with thousands of pages. Thin means there is nothing on the page a stored copy would carry: forty words, a photograph and a price, with the forty words repeated from a supplier's catalogue on nine other sites. That is not a quality opinion, it is a storage decision, and the honest question is what this page contains that no stored page already contains.

What is actually in the store

An indexed page is a copy in a store, not the page you wrote. Markup is gone, text is cut into tokens, and anything assembled by script after the fetch may or may not be there. Thurlby's specification tables loaded from a separate request and were missing from the stored copy for eighteen months, which is why the shop never appeared for any query naming a tile thickness.

In the store

What the stored copy contains versus the live page you wrote

Rendered HTML inspected in Search Console in 2026 shows tokenized text without markup and exposes script-loaded gaps.

Search Console URL Inspection (rendered HTML)
  1. 01Markup stripped, text cut into tokens
  2. 02Separate request missing from stored copy for eighteen months
  3. 03370 pages stored and never shown for anything
  1. 01Tokenized stored documentOnly the stored copy reaches the ranker.
  2. 02Script-fetched specification tablesPrevented Thurlby from appearing for any tile thickness query.
  3. 03Stored but unshown inventoryNo report splits weak stored pages from unsearched queries.

Inspect rendered HTML in Search Console instead of relying on how the live page looks in a browser.

Search Console's URL inspection shows the stored copy, free, and it replaced guessing from the absence of a page in results. As of 2026 the useful habit is to read the rendered HTML it reports rather than the live page: they are different documents, and only one of them is what the ranker sees.

What happens to a stored page once somebody types is what actually decides the order, and whether it got fetched in the first place is how a page is found in the first place. Both are inside the mechanism, before any tactic makes sense. Thurlby still has 370 pages that are stored and have never been shown for anything, and nothing in any report says whether that is the pages or the queries.

Alongside this

Above this