One event, not 125 faults

The situation

The ISP delivers each client as a Layer 2 service tunnel from a CPE at the facility back to an anchor router at the PoP. Monitoring showed each tunnel's running flag and counted the falses. Several times a week the count jumped from about 39 down to 125 down and back within hours, with the same single survivor every time. Technicians were being sent to sites that were fine, and the real 39 were lost in the noise.

What we did

We treated the tunnel flag as one witness, not the verdict. Every hour the collector also probes the management port of every CPE and reads the anchor's uptime and OSPF neighbours. A tunnel is down only when the flag and the facility probe agree. If the flag says down but the facility answers, it is suspect, which is a monitoring question, not a dispatch. If a region flips by more than a threshold in one sample while the anchor is healthy, the batch is tagged a correlated event with low confidence, and the report says so in one line instead of listing 86 outages.

While building it we found that the second region's anchor had been missing from the recon rotation, so 93 tunnels had been showing ten-day-old state. Adding it, with a longer timeout because that router carries a very large host table, cleared 28 false downs in one run.

The rule lives in one shared module and is applied in three places: the live dashboard, the hourly report, and the WhatsApp dispatch. They cannot disagree with each other.

What changed

Steady state is reported honestly at about 39 down, and every one of those has an unreachable CPE, so they are real. Surges are one event. Dispatch messages dropped to match. The raw per-tunnel data is one click away for anyone who wants to check the model's work.