Faragiri

Learn Crawling How search works

A Staffordshire salvage yard had 4,120 stock pages and a crawler that had never seen 900 of them, because nothing on the site pointed anywhere near them.

4Sections
4Min read
4Depth

A crawler finds a page by following a link to it or by reading it in a sitemap. There is no third way. Croxden Salvage, an architectural salvage yard in Staffordshire, listed 4,120 items on its site in 2026 and could not work out why a third of them never appeared anywhere, until somebody exported the internal links and found that 900 of the 4,120 pages had no link pointing at them from anywhere.

They were in the sitemap, which is why they were known about at all. A sitemap is a hint, not an instruction: it tells a crawler that an address exists and says nothing about whether the address deserves a fetch, and on a site of four thousand pages that difference decides which third gets seen.

The queue, and what keeps a page in it

Everything a crawler does is a loop. It takes an address off a queue, fetches it, reads the links on what came back, and puts those addresses on the queue. That loop is why links do the work: a page that is linked from three places enters the queue three times as often as one linked from none, and a page linked from nowhere enters it only because a sitemap put it there once.

The queue

The loop that decides which of four thousand pages gets seen

Nothing in this circle makes a judgement about quality; it is a queue fed by links, which keeps deep or unlinked pages waiting.

  • 01A link is followed
  • 02The address joins the queue
  • 03It is fetched
  • 04Its own links go back in
  1. 01A link is followedOr a sitemap entry is read once, which carries less weight.
  2. 02The address joins the queueThree inbound links mean three times as many chances to be queued.
  3. 03It is fetchedEvery three days near the home page, every 47 days four clicks deep.
  4. 04Its own links go back inKeeping the stock pages below it in regular rotation.

The loop says nothing about whether a fetched page is stored, which is a separate decision.

The rhythm is visible in Croxden's own crawl statistics. The pages linked from the home page were refetched every three days, and a stock page four clicks deep went 47 days between fetches. Same site, same server, same week. The only difference between those two numbers is how many links away from the front door each page sits.

What to change, and what not to

Clicks from the home page is the number to reduce, not pages. Croxden's instinct was to write more, and more pages on a site whose crawler is already 47 days behind makes the problem larger. What worked was three links: a page listing the twelve categories, a recently added block on the home page, and cross links between items from the same reclamation.

Reducing depth

Cutting fetch delays from 47 days to nine with three link paths

Croxden reduced click depth instead of writing more pages, moving deep stock closer to the home page.

Four clicks deep47 days between crawler visits
  1. 01Twelve-category directory pageGives crawlers one shallow index to all categories.
  2. 02Recently added home blockPuts fresh stock within one click of the front door.
  3. 03Same-reclamation cross linksConnects related items directly to each other.
Two clicks deepNine days between crawler visits

No extra pages were written and no server setting changed; only link paths shifted.

The deepest stock page went from four clicks to two and its fetch interval dropped to nine days. Nothing was written, nothing was submitted, and no setting was changed. A crawler has a budget on a big site and spends it on what is linked, so the whole of this work is about where the links point.

Where this stalls: spending the budget on nothing

Croxden's filters generated addresses. Every combination of material, period and price band produced a URL, and most of them returned an empty list, and there were about 1,600 of them being fetched.

Crawl budget stall

Why missing pages stay invisible while empty filters get fetched

Reports show what failed to appear, hiding the 1,600 empty filter combinations consuming daily crawl requests.

  • Missing stock pages in reportsThe visible gap site owners look at first.
  • Guessing more content is neededAdds more URLs into an already starved queue.
What reports highlight vs what consumes crawl requests
  • 1,600 empty filter URLsGenerated by material, period, and price combinations.
  • Fixed daily request limitSpent fetching blank lists instead of real salvage items.

Removing those 1,600 dead filter URLs freed crawl budget immediately for actual inventory.

Removing 1,600 dead filter URLs freed the fetches for real pages. That is the whole mechanism of what people call crawl budget, and at this size it is not an abstraction: a crawler making a fixed number of requests a day spent most of them on pages with nothing on them, because the site kept generating links to them and the crawler kept following links.

Doing this deliberately on a catalogue rather than by accident is search for sites with thousands of pages. The stall is that nobody looks at what is being fetched, only at what is missing. The missing pages are visible in a report and the wasted fetches are not, and the wasted fetches are the reason the missing pages are missing.

Reading the fetching itself

Search Console's crawl statistics report shows how many requests were made, what they returned and what kind of file they were, free, and it replaced reading raw server logs, which is still the more complete answer and needs somebody who can read them. The number to look at first is not the total. It is the split by response: Croxden's showed a fifth of all fetches returning empty result pages, which is what sent somebody to the filters.

Reading fetches

Checking response splits and sitemap rules before guessing

Crawl statistics and WordPress sitemap settings expose wasted fetches without reading raw server logs.

Post-crawl inspection checks

  1. YesCheck Search Console response splitRevealed a fifth of Croxden fetches hitting empty result pages.Yes
  2. YesReview Yoast post types and archives (2026)Keep filter URLs and unwanted archives out of the XML sitemap.Yes
  3. NoStart with raw server logsMore complete, but only useful if someone can parse them.No
  4. Not checkedAssume all 900 orphan pages will be keptLinking triggers a fetch; storing is decided by a separate system.Not checked

Search Console crawl statistics and Yoast sitemap controls are both free in standard setups.

Yoast SEO generates the sitemap and can be told to keep the filter URLs out of it, free in the version most WordPress sites run, and it replaced hand-written XML files that went stale the first time somebody added a page. As of 2026 the setting that matters is which post types and archives are included, and the default includes several that no site wants listed.

A page nobody links to is a page nobody has to be told about, and fixing that is sending authority where you want it with a much simpler motive than authority. What happens to a page after it has been fetched is a different decision made by a different system, at being stored, and why a crawled page may not be, and the whole loop sits inside the mechanism, before any tactic makes sense. Croxden still does not know whether the 900 will be fetched or dropped once they are linked.

Alongside this

Above this