The booking system for a chain of three climbing gyms went down on a Saturday morning and stayed down for 51 minutes. The alert that told the on call engineer arrived 34 minutes in, and it was a customer on the telephone. The server had not crashed. It had run out of disk, which stopped the database accepting writes, which the application reported as a booking error and nothing else.
Almost everything about running servers is contained in that gap between the failure and the noticing. A server is not a desktop that stays on. It is a machine somebody is depending on right now without being in the room, and the discipline is entirely about what happens when it stops.
The three numbers worth an alert, and the one that is not
Processor load is the number everybody graphs and it is rarely the one that ends a Saturday. The three that reliably matter are disk space, memory pressure and whether the service is answering at all.
A disk at 90 per cent is an alert; a disk at 100 per cent is an outage, and the distance between them can be an hour on a busy machine or a month on a quiet one, which is why the alert has to fire on the trend rather than only on the level. The climbing gym's disk had been over 90 per cent for nine days, and had it been noticed on any of them the repair would have been a five minute log rotation.
The check that would have caught it regardless is the one that asks the question a customer asks. Monitor the thing a user does, not the thing the machine reports: a request that creates a test booking every minute and alerts when it fails would have fired at minute one rather than minute 34. Machine metrics tell you why something broke. A synthetic check tells you that it broke, and those are different jobs that people routinely try to do with one tool.
Reading the machine when it is already unhappy
Two commands answer most of it, and both are already installed.
df -hprints every filesystem with its use as a percentage, and the one that is full is rarely the one you expectedjournalctl -p err -S -1hprints only the errors from the last hour
df -h reports per filesystem, so a full disk can be one directory. Machines are frequently configured with separate volumes, and a log directory filling up while the main disk sits at 40 per cent is the ordinary case rather than the exception. Looking at a single overall figure is how a full disk stays invisible while a graph looks healthy.
The other half of uptime is the change you made
The largest single cause of servers becoming unavailable is not hardware and not attack. It is a change, deployed by the people who own the machine, on a working day. That is not an argument against changing things; it is an argument for changing them in a way that can be undone, and for not doing it at four on a Friday.
The practical version is that a service should start clean from its configuration file with one command, and that the configuration file should be in version control. If restoring last week's behaviour means remembering what was edited, the machine is not recoverable, it is merely still running. The software that actually answers requests, and the configuration that decides how, is the software that answers HTTP.
One Saturday
The failure took a second and the noticing took 34 minutes
Five moments in one outage. Only the last two involved anybody touching the server.
- Nine days outDisk passes 90 per centNo alert configured on this volume, and the overall figure still reads comfortable
- 09:12The disk fillsThe database stops accepting writes and the application reports a booking error
- 09:46A customer telephones34 minutes in, and this is the monitoring system that actually worked
- 09:58df -h names the volumeTwelve minutes to find it, because the machine's own graphs were of the wrong number
- 10:03Logs rotated, writes resumeFive minutes, which is what the repair would have cost on any of the nine days
The spacing is not proportional and the shape is the claim: the repair was five minutes and the outage was 51, and everything between was finding out.
What makes a server different
The differences from a desktop are concrete rather than philosophical: it has no screen, it is expected to survive a power cut with its data intact, it is patched on a schedule somebody decided rather than when a dialog appears, and several people have administrative access to it. Each of those changes how it is set up, and together they are the substance of the difference from a desktop, in practice.
The services people notice within minutes of failure are mail and shared calendars, and they carry a reputational cost that a slow website does not, which is the argument for treating the services people notice immediately when down as a separate tier with tighter alerting. Where the machine physically lives, and who is responsible for the layer beneath it, is where a site actually lives, and that decision determines what you are allowed to fix yourself at two in the morning.
All of this is the same discipline the rest of the machines, the networks between them, and keeping both standing applies to a machine in front of you, moved to one that nobody is looking at, which is the only real difference and the reason the noticing is the subject.
Where people stall
The stall is alert fatigue, and it arrives about six months after the first monitoring system. Somebody enables the default alerts, receives forty a day, learns to ignore the channel, and the one that mattered arrives into a room that stopped reading. An alert nobody acts on is worse than no alert, because it manufactures the belief that the system is being watched. The correct number of alerts on a small estate is a handful a week, and getting there means deleting most of what a monitoring tool suggests, which feels like negligence and is not.
The second stall is the Windows estate, which is administered through entirely different tooling and is routinely treated by Linux administrators as a lesser subject rather than a different one. It is the majority of corporate server work, and the Microsoft server estate is a full profession.
What to open, in 2026
For metrics, Prometheus with Grafana is the default and both are free and open source. They replaced Nagios and its generation, which checked whether a thing was up rather than recording how it behaved over time, and the difference matters exactly in the case above: a level tells you nothing that a trend would not have told you nine days earlier.
For logs, journalctl is already on the machine and is enough for a small estate. Reach for a central log system when there are more machines than you can log into, and not before, because the honest cost of running one is a machine of its own.
For the alerting itself, the free tiers of the hosted on call tools handle a rotation for a small team, and the thing worth paying attention to is not the tool but the runbook: a page that says what to check, in order, for each alert, written by the person who fixed it last time. That is what watching before it falls over is ultimately about, and it is the part that is not bought.
What none of this covers is what happens when the machine is not yours at all and the layer under it is rented by the hour, which is the topic beside this one.