A crawler slows down when a server is slow, and speeds up when it is not. Hexton Hardware sells 12,000 lines of ironmongery online, and in January 2026 its server was answering in about 1.4 seconds and being fetched around 900 times a day. 12,000 products at 900 fetches a day is a fortnight per pass, which means a price changed on a Monday was seen a fortnight later.
After caching was put in front of the category pages the response time fell to about 0.3 seconds, and fetches a day went from 900 to 4,100 over six weeks. The weekly averages run 900, 1,200, 1,900, 2,600, 3,400 and 4,100. Nothing was asked for and no setting was changed on the search side; the crawler simply took more, because taking more had stopped being expensive.
Which number this is about
Server response time is the number that matters here, not the page score. The two are related and they are not the same: a page can render slowly because of a large image while the server answers in 200 milliseconds, and a crawler does not care about the image because it is not drawing anything.
Server bottleneck
Why caching category pages freed the whole catalogue
The crawler walks category links to reach products, so slow stock queries on category pages held back every product fetch behind them.
- Crawler requests
- Category links
- Catalogue passes
- Product pages
- Price changes
- Stock updates
A crawler does not render large images; it waits on the server response time, and caching category pages removed the queue.
Caching the category pages did more than anything on the product pages, which is the opposite of the instinct. The category pages are what a crawler walks through to reach everything else, they were the slowest thing on the site because each one ran a stock query across 12,000 rows, and every product fetch was waiting behind them.
The other way to be crawled less
A 500 or a timeout makes a crawler back off hard, and it backs off for longer than the fault lasts. Hexton had a Friday afternoon in February when the database was locked for about twenty minutes and returned errors, and the fetch rate did not recover for four days afterwards.
Error backoff
Twenty minutes of 500 errors slowed crawling for four days
Crawlers back off hard on timeouts and server errors, and the reduced fetch rate lasts far longer than the fault itself.
- Fault beginsDatabase locksRequests hit timeouts and 500 responses instead of pages.
- 20 minutesLock clearsThe server answers normally again, but the crawler has stepped back.
- Days 1 to 3Crawl stays throttledFetch volume remains low because the site signalled instability.
- Day 4Rate recoversDaily fetches return to normal four days after a twenty-minute outage.
Returning errors during a release tells the crawler the site is unreliable, turning a brief deploy fault into days of slower indexing.
That is worth knowing before a deployment. A site that returns errors during a release is not merely offline for ten minutes; it has told a crawler that it is unreliable, and the reduced rate outlasts the release by days.
Where this stalls: doing it on a small site
This only matters on a site with more pages than the crawl can cover. At 90 pages the whole subject does not arise: a crawler covers the site in a morning at any plausible speed, and a firm reading a crawl budget article and optimising its server has spent money on a constraint it does not have.
Site size test
Divide total addresses by fetches a day before tuning servers
Crawl budget only bites when a site has more addresses than daily fetches can cover in a reasonable pass.
- Yes (days per pass)Cut server response timeHexton Hardware needed fourteen days per pass across 12,000 products.
- Cache slow category queries
- Prevent 500 errors during releases
- No (hours per pass)Skip crawl budget workA joinery with 90 pages is covered in under an hour at any plausible speed.
- Ignore big-site crawl advice
- Spend budget on content or checkout
Advice written for million-page catalogues travels further than the constraint; simple division shows whether the bottleneck exists.
The stall is that the advice travels further than the situation. Crawl budget is written about by people working on sites with millions of pages, it is read by people with two hundred, and the honest test is arithmetic: divide the number of addresses by the fetches a day and ask whether the answer is a problem. At Hexton it was fourteen days. At a joinery with 90 pages it is under an hour.
What to read, and what it shows
The crawl stats report shows response time against fetches on the same chart, which is where the relationship becomes visible rather than theoretical. It is in Search Console, free, and it replaced reading raw server logs, which is more complete and needs somebody who can read them. The two lines at Hexton move in opposite directions across February and March, which is the whole of the evidence on this page.
Crawl rate
Pages fetched a day, across the six weeks after caching went in
Nothing was requested and no setting was changed on the search side; the rate rose because fetching had stopped being expensive.
Pages fetched a day, Hexton Hardware, weekly averages from Search Console, January to March 2026.
- 900W1
- 1,200W2
- 1,900W3
- 2,600W4
- 3,400W5
- 4,100W6
One site, one change, six weeks. The dip after the February error afternoon is not visible at weekly resolution, and it took four days to recover from twenty minutes of faults.
The report also splits fetches by response code and by file type, and both are worth a look once. At Hexton a fifth of all requests were for images, which is normal, and 4 per cent were 404s from a category structure abandoned in 2024, which is not.
What those 404s should have been pointing at is moving a page without losing what it earned, keeping the crawler off the filter addresses entirely is what to let bots see, and what to keep out, and what happens to a page once it has been fetched is being stored, and why a crawled page may not be. All of it is the far end of how fast it loads, and what that is worth in search, and it is the only part of speed that is about a machine rather than a person.