A path-health job that runs every five minutes started returning UNKNOWN for every path, all at once. Run by hand from a terminal, the same script was green. Nothing on the network had changed.
The script shells out to ping and route. On macOS those live in /sbin, which is on an interactive shell's PATH and not on cron's. The scheduler was running the script, the script was calling a command that did not exist in its world, and the error handling did the right thing: it refused to claim a status it could not measure.
What we do now
- Any script that a scheduler runs calls tools by absolute path, or sets
PATHexplicitly at the top. - A result of UNKNOWN is treated as a monitoring fault, and it pages the person who owns the monitoring, not the person who owns the network.
- The dashboard shows a red "sensor" state separately from "down", so an outage of the measurement never reads as an outage of the thing measured.
The first rule is obvious. The other two are the ones that would have caught it in five minutes instead of a day.