Faragiri

Learn Sitemaps Technical

A Cheshire rug importer had 2,400 addresses in its sitemap and 1,700 pages on its site, and the difference was every page it had ever deleted.

4Sections
3Min read
4Depth

A sitemap says an address exists, and asks for nothing. Runcorn Rugs imports and sells rugs from Cheshire and had 2,400 addresses listed against 1,700 pages that exist, a gap made entirely of pages deleted since 2023 and never removed from a file nobody looked at.

Every address in it should return 200 and be indexable. That is the whole specification and both halves get broken: 700 of Runcorn's addresses returned 404, and another 90 returned 200 while carrying a noindex, which is a site listing pages it has decided not to publish.

What it is for, given that it asks for nothing

Submitting it does not make anything be indexed. It makes addresses known, which matters only for pages nothing links to, and that is a smaller set than most people assume. On a well linked site the sitemap tells a crawler almost nothing it could not find by following links.

Coverage

What the submitted-versus-indexed gap revealed at Runcorn

A sitemap earns its keep in the coverage report rather than by forcing pages into the index.

Runcorn Rugs URL counts

  1. Submitted URLsAddresses declared in the sitemap for discovery and reporting.1,700
  2. Indexed pagesPages actually kept after crawling and evaluation.940
  3. Fetched and dropped productsProduct pages crawled but excluded from storage, not a sitemap bug.600

Comparing 1,700 submitted against 940 indexed exposed 600 dropped product pages immediately.

Where it earns its place is the report rather than the crawl. The coverage gap between submitted and indexed is the useful reading, and at Runcorn 1,700 submitted against 940 indexed pointed straight at 600 product pages that were being fetched and dropped, which is being stored, and why a crawled page may not be rather than a sitemap problem at all.

The two contradictions

An address in the sitemap and disallowed in robots.txt is two instructions, and a site giving two instructions is a site hoping the more optimistic one wins. Runcorn listed 210 filtered addresses in the sitemap while blocking the same parameter in robots.txt, which came from two people solving two problems eight months apart.

Hygiene

Two contradictions that make crawlers ignore a sitemap

Listing blocked URLs or faking nightly modification dates tells crawlers the file cannot be trusted.

Clean sitemap signals

  1. NoList URLs blocked in robots.txtRuncorn listed 210 filter addresses while disallowing the same parameter in robots.txt.No
  2. NoStamp every URL with today's dateClaiming 1,700 pages changed last night trains crawlers to ignore lastmod completely.No
  3. YesExclude disallowed parameter pagesKeep blocked filter combinations out of the sitemap so rules never conflict.Yes
  4. YesUse real modification datesConfigure the generator to record only genuine page updates in lastmod.Yes

Conflicting directives usually happen when two people solve two problems months apart without checking the sitemap.

Lastmod is only useful if it is true. A generator that stamps today's date on every address every night is telling a crawler that 1,700 pages changed last night, which is a claim that gets believed once and then ignored. Runcorn's plugin did exactly this, and switching it to the real modification date is a setting rather than a fix.

Where this stalls: the file nobody owns

Generate it, never maintain it by hand. A hand written sitemap is correct on the day it is written and wrong within a fortnight, and the reason Runcorn had 700 dead addresses is that the file was made by an agency in 2023 as a one off deliverable.

Ownership

How a hand-made sitemap drifts silently over three years

A static file breaks no pages and raises no errors, so stale entries sit unnoticed for years.

  1. Day 1 (2023)Agency delivers a static fileA hand-built sitemap matches the live site on the day it is created.
  2. Within a fortnightEdits leave the file behindDeleted or moved pages stay listed because nobody updates the XML manually.
  3. Three years later700 dead URLs sit unlisted as errorsNo page breaks; the only symptom is a noisy coverage report.
  4. RemedyGenerate dynamically from live pagesAutomated generation keeps listed addresses tied to what currently exists.

Generate sitemaps automatically so maintenance never depends on somebody remembering a static file.

The stall is that nothing ever reports a stale sitemap as a problem. It is not an error, no page breaks, and the only symptom is a coverage report that is harder to read than it needs to be. It sat wrong for three years at a shop with somebody actively working on search.

The mechanics worth knowing once

50,000 addresses or 50 megabytes per file, then use a sitemap index, which is a sitemap of sitemaps and is what every generator produces by default. Runcorn does not need one at 1,700 pages and has one anyway, split into four files of 1,240 products, 38 pages, 410 posts and 210 filter addresses, which is genuinely useful for a different reason: the coverage report can then be read per type, so a problem with product pages does not hide inside a total that includes blog posts.

Sitemaps

One index, four child files, and what each isolates

Splitting by type is worth doing for the reading rather than for the 50,000 limit, so product faults do not hide inside a site total.

sitemap_index.xml (up to 50,000 URLs or 50 MB per child)

  • product-sitemap.xml1,240 addresses, isolating the 600 dropped products from blog totals
    • 700 dead addressesDeleted URLs needing redirects rather than silence
  • page-sitemap.xml38 addresses
  • post-sitemap.xml410 addresses
  • filter-sitemap.xml210 filter addresses separated for inspection
    • 2,400 discovered vs 1,700 liveSearch Console exposes inflation across the combined child files

Yoast SEO builds the index and child files automatically; as of 2026, choose carefully which post types and taxonomies stay included.

Yoast SEO generates the index and the child sitemaps, free in the version most WordPress sites run, and it replaced hand written XML files that went stale on the next page added. As of 2026 the settings that matter are which post types and taxonomies are included, and the defaults include archives that most sites do not want listed.

Search Console is where the file is submitted and where the reading happens, free, replacing counting pages by hand and believing the total. Its sitemaps report shows the count discovered per file, which is the fastest way to notice that a file has 2,400 addresses in it on a site with 1,700 pages.

What should be kept out of the file entirely is what to let bots see, and what to keep out, the 700 dead addresses each need somewhere to point, which is moving a page without losing what it earned, and the 600 dropped products are pages that dilute the ones that matter. Every one of those is a decision about a file rather than about a page, which is the layer that decides whether any of the rest counts.

Alongside this

Above this