What monitoring should actually cover on a small server estate

Small estates fail in one of two directions. Either nothing is monitored and you find out from a client, or someone installed an agent on every box, enabled the default alert set, and now nobody reads the alerts. The second is worse.

One rule does most of the work

Page on symptoms. Record everything else.

A symptom is something a user would notice. A cause is the reason underneath: the disk filled, a process died. Causes are many and symptoms are few. Twenty faults produce “the site is returning 500s”, so you alert on the 500s regardless, and every cause-based alert after that is a second notification for an incident you already knew about. Those assumptions also go stale: the alert you wrote in 2023 for a queue that no longer exists still fires.

For a team of three, over-monitoring is harder to fix than under-monitoring. Once people ignore the alerts channel, adding a good alert changes nothing.

The CPU alert is the classic mistake

Nearly every estate we inherit alerts on CPU above 80%, and it has usually been muted. A box pinned at 95% while serving every request inside target is fine, and one at 30% throwing errors is not. What top reports is only the time the CPU was not running the idle thread, which lumps together doing work and stalling on memory.

The Prometheus node-mixin settles the argument: NodeCPUHighUsage fires at 90% for fifteen minutes at severity info, while saturation alerts rank higher. Load average per core, disk queue depth, and major page faults over 500 a second, which is the metric that actually says a machine is thrashing. Keep CPU and memory on a dashboard, not in the alert set.

Start outside the box

The highest-value check runs somewhere you do not control. An agent can tell you a process is alive. It cannot tell you the domain expired, the DNS delegation broke, or the host is unreachable. Configure an external check that fetches a real page over HTTPS and looks for a string that appears only when the application is working, rather than a TCP connect or a health endpoint that returns OK while the database is down. This is also the check allowed to wake someone up, because it only fires when something is broken from outside.

Certificates: the advice you remember is out of date

Let’s Encrypt stopped sending expiry notification emails in June 2025, so a renewal that quietly breaks now tells nobody. Lifetimes are collapsing too: the CA/Browser Forum maximum dropped to 200 days in March 2026 and reaches 47 in 2029. Thirty days’ warning on a 47-day certificate fires with two thirds of its life left, which trains people to ignore it.

Set the warning as a fraction of the lifetime instead, and watch three things rather than the expiry date: days left on the certificate actually being served, probed from outside; renewal failures from the ACME client; and unexpected issuance in Certificate Transparency logs.

Disk: predict the fill, then keep a floor

A static threshold fails both ways. A partition sitting at 85% for two years gets muted; one going from 20% to full in an hour trips far too late. Ask instead whether the disk is going to be full, and how soon:

predict_linear(node_filesystem_avail_bytes[6h], 4*3600) < 0

Guard that with a percentage floor, which suppresses the daily sawtooth from log rotation and backups, and keep a plain 5% threshold underneath. Prediction buys lead time; the floor is the guarantee.

Backups: monitor restores, not backups

A backup job reporting success proves the job ran. It does not prove the archive is readable, that the dump inside is consistent, or that there is a machine to restore onto. The 3-2-1 rule became 3-2-1-1-0 because the rest of it was satisfied by people who had never tested a restore.

In order of value: a heartbeat on the job, so a backup that silently stops running becomes an alert rather than a discovery; integrity verification on a schedule, where restic check --read-data-subset 10% reads blobs rather than metadata; and a restore drill on the calendar. Only the third proves anything about recovery.

Who watches the watcher

Every monitoring system fails silently, and silence is indistinguishable from health. If the monitoring box dies or the webhook starts returning 404, you get no alerts, which looks precisely like nothing being wrong. The fix is a dead man’s switch: an alert that always fires, routed to a service outside your infrastructure that expects to hear from it and shouts when it does not. Give it a different notification path from your normal alerts, or one broken webhook takes out both.

What this looks like in practice

For three to twenty servers with nobody whose job is operations: external checks hosted off your infrastructure, certificate expiry as a fraction of lifetime, heartbeats for every backup and scheduled job, host metrics with a maintained alert set rather than one you invent, and a dead man’s switch terminating elsewhere.

What stays off the list: container restarts, latency at more than one layer, anything with an automated remediation, and anything nobody has acted on in six months. Deleting alerts is maintenance, not neglect.

The result fits on a page, whoever is on call can understand it, and every alert in it is one a person can act on.