The biggest supplier in three years of a food co-op's deliveries turned out to be the wrong one, and the whole error was a spelling. Ruth Adeyemi had been negotiating the following year's terms with the second largest for a fortnight before anybody counted properly.
The export from the till was 1,842 rows: date, supplier, kilos, price. Ashgrove Farm appeared in it as Ashgrove Farm, Ashgrove Farms, ASHGROVE and Ashgrove Farm Ltd, which the till had been perfectly happy to accept because nothing in a text field says two strings are one supplier. A name spelled four ways is four suppliers until somebody merges them.
What the merge did to the answer
Sorted as exported, Bridge End Growers led on 9,410 kg and Ashgrove Farm sat second on 4,980 kg. Add the other three spellings, 2,760 kg and 1,905 kg and 1,240 kg, and Ashgrove has delivered 10,885 kg, which puts it first by a margin nobody had seen. Every other supplier in the table stayed exactly where it was.
The check that would have caught it takes one line and about four seconds. Count the distinct values of a column before trusting any total built on it: a supplier column with 34 distinct names in a co-op that deals with 29 suppliers is a finding, and it is a finding you get before any analysis rather than after a fortnight of it.
What arrived, and what could be used
Of the 1,842 exported rows, 96 were test deliveries from the migration to the current till, all dated the same Tuesday. That left 1,746. Another 48 had no weight at all, and blank is not zero: a delivery with no weight recorded is a row you know nothing about, and treating it as a zero kilo delivery drags every average down. That left 1,698.
Then the dates. 61 rows had come out of an older export in a format that reads both ways, so 03/04 could be the third of April or the fourth of March. An ambiguous date moves a season of rows into the wrong quarter, silently, and unlike a bad number it never looks wrong. Those 61 had nothing to break the tie with, so they were dropped, leaving 1,637 rows that mean one thing each.
Two hundred and five rows removed out of 1,842, and none of the removals is arguable once written down, which is the actual product of a cleaning session. Write down what you changed, or the answer is not reproducible: the same export run through the same eight decisions gives the same 1,637 rows next quarter, and run through somebody's memory of them it does not. That is the whole argument of the eighty percent of the job nobody photographs.
The order
Four links, and the one that is usually drawn shut
None of these matters more than another, and the answer is only as good as whichever one was skipped.
- Getting itWhat somebody chose to record decides what can be asked
- Cleaning itThe link most courses draw closed and most jobs spend the week on
- Analysing itMedians beside means, and a look at whatever disagrees
- Showing itWhere an honest analysis can still mislead, by its axis
The open link is not the hardest one. It is the one whose failures do not announce themselves, which is a different property and a worse one.
One row can move a mean and not a median
The average delivery across the cleaned rows came out at 41 kg, which nobody at the co-op recognised. One row was the reason: a delivery entered as 8,275 kg where the invoice says 82.75 kg, a decimal point lost on a Friday. Remove it and the average is 36 kg.
The median was 32 kg both times. A median survives one absurd row and a mean does not, which is why the two are worth printing side by side on any table nobody has cleaned yet: where they disagree badly, the disagreement is a pointer at the rows to go and look at. Reading a table this way is turning a table into an answer, and most of it is this unglamorous.
The axis is part of the claim
The chart that went to the co-op's members showed monthly kilos with the vertical axis starting at 30 tonnes. The year's rise was about a tenth, and on that axis it looked like the co-op had roughly doubled its trade.
Nobody intended it; the plotting library chose the axis, as plotting libraries do, to fill the space. A truncated axis is a claim, and the claim is usually not the one intended, so the rule is to decide the baseline yourself and say what it is. Making the finding visible is the last link and the one where an honest analysis can still mislead.
Four links, and the one drawn open
The order is the pipeline. Where it comes from before it is useful first, because the shape of what you can ask is decided by what somebody chose to record. Then cleaning, then the analysis, then the chart, and what the analysis is actually done in alongside all four rather than before them.
Taken backwards, which is how it is usually taught, a reader learns plotting and statistics against tidy teaching sets and meets a real export for the first time in a job. Every teaching set has already had its 205 rows removed by somebody, and that removal was the lesson.
Before and after one merge
One row moved, and it was enough to change the answer
Merging four spellings of a single farm changed one supplier's total and left every other row exactly where it was.
as exportedafter the spellings were merged
- Ashgrove FarmFour spellings, one farm, three years4,980 kg10,885 kg
- Bridge End GrowersSpelled one way throughout. Nothing moved9,410 kg9,410 kg
- Peckham Rye DairyNothing moved here either, which is the point of drawing it3,120 kg3,120 kg
Two rows that did not move are doing as much work here as the one that did. A before and after with only the changed row on it is an advertisement for the change.
The stall is a clean table that was never questioned
People stop here at the point where the file loads without errors. Nothing complains, every column has a sensible type, the totals add up, and the table now looks like the teaching sets. It does not feel like stopping, because the work genuinely got harder to do and the errors genuinely went away.
What is missing is the habit of asking the table what it cannot know. Ruth's 1,637 rows cannot tell her about a supplier the co-op stopped using before the current till, cannot distinguish a delivery that was refused from one that never came, and cannot say a word about price per kilo, because the price column is what was invoiced and the kilos column is what was signed for on the day. Every one of those is a sentence about the collection rather than about the data, and none of it is visible in the file.
What to open in 2026
DuckDB runs SQL directly over CSV and Parquet files on a laptop with no server to install, free and open source as of 2026, and it replaced loading a file into a database before you could ask it anything. The distinct-count check above is one query against the exported file, before it has been imported anywhere.
OpenRefine is the one built for exactly the Ashgrove problem, free and open source as of 2026: it clusters values that are nearly the same and offers the merge, which replaced sorting a column by hand and squinting at it. It also keeps the list of operations you applied, which is the reproducibility point made into a file rather than a resolution.
pandas remains the default for the analysis itself, free and open source as of 2026, and it replaced a spreadsheet with a formula copied down 1,842 rows and one row where somebody stopped copying.
Which leaves what none of the cleaning can fix. Every decision above was made about rows that already existed, and the interesting question is what was never recorded: the co-op has no row at all for a delivery that was offered and turned down, so no amount of work on this file will ever say what it declined to buy. That is a question about collection rather than analysis, and it is the same question underneath models learned from data rather than written by hand, where an absent row becomes an absent case. It is also why measuring it, so decisions are not guesses spends most of its time on what to record rather than on what to compute.