Faults confirmed before anyone is sent

What we built

A service counts as down only when two independent checks agree: the service's own state, and a probe of the equipment at the client's site. If they disagree, the service is suspect, which is a question for monitoring rather than a reason to drive out. When many services drop at once while the router upstream of them is healthy, they're reported as one event with a confidence level, not as a page of alarms.

The same rule feeds a report that runs at the top of every hour. It compares each run with the last, so an issue is new once and ongoing after that, and it keeps its age. It reads the network's structure, so when a site upstream is down, the sites behind it are listed as consequences under one cause. Each fault is sorted by the fix it needs: a remote change, a visit, or nothing until the upstream fault is cleared. A local language model writes the summary from a short digest, so the data stays in-house.

How it holds up

One shared rule feeds the dashboard, the hourly report and the dispatch messages, so they can't disagree. If the model is offline, the tables still go out. The report checks twice before it calls a node down, needs two runs in a row before an issue counts as new, and holds the email when it can't see the network properly. It has run every hour since June 2026.