Faragiri

Learn How search works SEO · Crawling, Indexing, Ranking and 1 more

A Leicester printer's page was fetched on one date, stored on another, and ranked on a third. Nearly every SEO argument is about which one failed.

4Branches
4Nodes below
3Depth

Three systems stand between a page and a result. A crawler fetches, an index stores, and a ranker orders, and they run at different times. Marrowbone Press, a letterpress printer in Leicester, put a page about wedding invitations online on 4 February 2026. It was fetched on 6 February, stored on 19 February, and first appeared for anybody's query in the middle of March.

Those three dates are the mechanism. Nearly every question that gets called a ranking problem is really a question about which of the three a page failed at, and the three fail for unrelated reasons and are fixed by unrelated people.

What each of the three actually holds

The crawler holds a queue of addresses and whatever came back over HTTP. It decides nothing. It fetches, reads the links it finds, and puts more addresses in the queue.

One page, four weeks

The four things that happen to a page, in the only order they can happen in

Drawn as a staircase rather than a list because none of these can be taken out of turn: nothing is ordered that was never stored.

  1. 01Fetched6 February. A queue, a request, and a copy of what came back.
  2. 02Processed and stored19 February. Tokens pointing at documents, not the page anybody wrote.
  3. 03Matched to a queryOnly when somebody types. The index supplies the candidates.
  4. 04Ordered against the restMiddle of March, and computed again from scratch every time.

The staircase says nothing about the gaps. Two days to the first step and thirteen to the second, at one small site, on one page, once.

The index stores a processed copy, not the page you wrote. Markup is stripped, script is run or not run, the text is cut into tokens, and what is kept is a set of entries pointing from a word to the documents that contain it. The page as a reader sees it is not in there at all, which is why a paragraph that a browser assembles after the fetch can be missing from the stored copy while looking perfectly present on screen.

The ranker holds nothing about the page until somebody types. There is no stored score per page; the order is computed for one query at a time, from the candidates the index can supply for that query. A page does not have a ranking. It has a position, for one query, at one moment, for one person in one place.

Crawled and not indexed is not a crawling problem

Search Console separates the two, and the two lines mean opposite things. Crawled, currently not indexed says the fetch worked and the storing was declined: the page was read and judged not worth keeping. Discovered, currently not indexed says it was never fetched. Crawled and not indexed is a different report line from not crawled, and the work each one calls for shares nothing.

Search Console lines

Two index exclusions that require opposite fixes

Marrowbone had thirteen pages in the first column and forty-one in the second; treating them as one problem wastes weeks.

Crawled, currently not indexedThirteen pages at Marrowbone
Discovered, currently not indexedForty-one pages at Marrowbone
Fetch statusCrawler fetched and read the HTMLURL known, never fetched
Why it stoppedIndex declined to store itCrawler never visited the address
Root problemPage judged not worth keepingNo internal links pointed to it
Required workRewrite or enrich the pageAdd internal links so bots find it

A crawler cannot value what it has not fetched, and rewriting an unlinked page changes nothing until a link reaches it.

Marrowbone had thirteen pages sitting on the first line and forty-one on the second. The thirteen needed writing. The forty-one needed a link from somewhere, because nothing on the site pointed at them and a crawler cannot want what it has not been shown.

The disallow that hides the noindex

340 old proof pages sat under a proofs directory and should never have been listed. Somebody added a noindex tag to every one of them and a disallow line to robots.txt on the same afternoon, on the reasonable theory that two locks hold better than one. They cancel each other. A disallow in robots.txt stops the crawl, so the noindex on the page is never read, and a page blocked in robots.txt can still be listed, with a URL and no description, because the address was found in a link somewhere else and nothing was ever fetched that could say otherwise.

340 proof pages

How robots.txt Disallow breaks the noindex removal chain

Putting two locks on the same directory cancels the inner one because the crawler stops before reading the page tag.

Allow the crawl first, let noindex clear the index, and add Disallow afterwards only if crawl budget demands it.

The order that works is the opposite of the intuitive one. Allow the crawl, let the noindex be read, wait for the pages to fall out, and only then disallow if there is a crawl reason to. Marrowbone took the disallow off on 2 March 2026 and the proof pages left the index over the following five weeks.

Where this stalls: one word for three systems

The stall is vocabulary. A page is not ranking, and that sentence covers a page never fetched, a page fetched and not stored, and a page stored and sitting twelfth. Three failures with nothing in common, described by one complaint, and the complaint sends everybody to the writing because the writing is the part that feels like work.

Diagnosis before rewriting

One complaint on the surface, three unrelated system failures below

Saying a page is not ranking sends everyone straight to editing copy before checking whether the page was ever fetched or stored.

  • 'The page is not ranking'Single catch-all phrase that hides which system actually stopped
A forty-second URL Inspection check separates these
  • Never fetched by crawlerUnlinked or blocked; rewriting copy changes nothing
  • Fetched, declined by indexRead and judged too thin or redundant to store
  • Stored, sitting twelfthIndexed and eligible, just outranked for that query

Checking the index first settles roughly a third of copy arguments before anyone opens an editor.

The tell is an argument about the wording of a page that nobody in the room has confirmed is in the index. That check takes forty seconds and it ends about a third of these arguments before they start.

What the results page is made of in 2026

The query letterpress wedding invitations leicester returns a generated answer of four sentences naming three sources, then the blue line, the breadcrumb, the snippet, and a box of four adjacent questions. Being named in the answer is a citation and a citation is not a position, which is a genuinely new distinction and the reason the fourth cluster below this one exists.

One query, 2026

What a results page is actually made of now

The first thing on it is not a result, and being inside it pays differently from being under it.

  1. 01A generated answer, four sentences, naming three sources
  2. 02Marrowbone Press | Letterpress Wedding Invitations
  3. 03marrowbonepress.co.uk / invitations / wedding
  4. 04Hand-set metal type on a 1962 Heidelberg, printed in Leicester.
  5. 05Four adjacent questions
  1. 01The answer panelBeing named here is a citation, and a citation is not a position.
  2. 02The blue lineRewritten by the engine when it judges a heading on the page a closer fit.
  3. 03The breadcrumbBuilt from the address or from structured data, and never from the menu.
  4. 04The snippetTaken from the page itself whenever the description does not answer the query.
  5. 05The question boxEach one a query the engine believes sits next to this one.

Drawn from a spec rather than a screenshot, so it is still readable when the layout changes, which on this query it has twice since 2024.

Google Search Console's URL Inspection reports which of the three stages a given address has reached, including the stored copy the ranker will actually see. It is free, and it replaced searching for your own page and guessing from the absence. Bing Webmaster Tools is also free and worth having for one reason: it replaces the assumption that every engine sees the same thing, and where the two disagree, the disagreement is usually the crawl.

Neither one shows the generated answer being assembled. Marrowbone can see that the invitations page is indexed and can see the answer naming a competitor, and nothing available in 2026 connects those two facts.

The four parts of the mechanism

Getting fetched at all is how a page is found in the first place, and the forty-one unlinked pages are what its absence looks like. Being kept is being stored, and why a crawled page may not be, which is the thirteen.

Four layers and a control rule

How the full search mechanism stacks up

Every outcome on the results page depends on the layers underneath it, while timing between removal and re-indexing stays opaque.

  1. Generated answer citationDraws on the index like links do, but rewards a citation rather than a position.Answer panel
  2. Query-time orderingComputed fresh for one query, one moment, and one searcher; stored nowhere.Ranker
  3. Processing and storageDecides what stays; thirteen crawled pages were dropped and 340 proofs took five weeks to exit.Index
  4. Discovery and fetchingFinds linked addresses; forty-one unlinked pages never reached this stage.Crawler
  5. Crawl and index directivesrobots.txt and noindex govern whether any layer above is allowed to operate.Control layer

No tool reports how long a bulk removal will take; Marrowbone's 340 pages cleared one by one over five weeks.

The order itself is what actually decides the order, computed per query and stored nowhere. Being named above all of it is being cited when the answer is generated, and it runs on the index like everything else while paying out differently.

The robots.txt line and the noindex tag both belong to the layer that decides whether any of the rest counts. What the four sentences in the answer panel were answering, and whether anybody was going to click anything afterwards, is finding what people type, and what they mean by it.

The five weeks it took the proof pages to leave is the number nobody at Marrowbone can explain. Nothing published says how long it should take, the pages went one at a time rather than together, and whether the next 340 would take five weeks or five months is not knowable from anything the shop can see.

Inside this, 4

Alongside this

Above this