An hour on the desk, every hour

The situation

The operator had monitoring but no narrative. Alerts arrived one at a time, a high-site outage looked like forty client outages, and nobody could say how long a given site had actually been down. The ask was a report a person could forward.

What we did

Collection is read-only: ICMP and TCP-connect to every backbone node, a management-port ping per facility, and RouterOS /print over the API for interface counters, CPU and memory on the spine routers. Radio signal and link capacity come from the Ubiquiti controller.

A ledger diffs this hour against the last, so an issue is new once and ongoing after that, and it carries its age across runs. Topology correlation reads the network tree: when a high-site is down, its downstream clients are symptoms, so the report names the root and groups the rest under it. Each down service is classified by where the chain breaks: config (the CPE answers but the tunnel is down, usually a tunnel-ID mismatch, fixed remotely), path or power (the CPE is unreachable, so someone drives), or cascade. A 30-day availability figure per service accumulates from the hourly samples.

The prose is written by a local language model from a compact digest and a short description of how the network works. If the model is offline the tables still render; the data never depends on it. Three guards stop it crying wolf: ping twice before calling a node down, require two consecutive runs before an issue is new, and hold the email entirely when the collector itself is degraded because the VPN or the core routers are unreachable.

What changed

The operator gets one email an hour with the root cause first and a table of sites needing attention, each with its fix type. The archive of hourly snapshots became the input for the next layer: attributing physical causes such as power, fibre cuts and theft by fusing counter resets with the operator's own WhatsApp history. Reboot detection from backwards counters is in; the cause engine is in progress.