Our recon tool asks each core router a list of /print questions over the RouterOS API and stores the answers by key. For months it paired replies to commands by position: first reply belongs to the first command, and so on. That worked until a router was upgraded and its bridge host table grew to about 204,000 rows, which takes 31 seconds to stream. Our per-command timeout was 20 seconds.
The tool gave up on that command and moved on. The router did not. Its late reply was still sitting in the socket, so the next command's answer was read as the one after it, and every key from there shifted by one. The service tunnels ended up filed under the VPLS key, the inventory saw zero tunnels for the region, and the dashboard reported 93 sites as missing rather than down, which is at least honest.
The fix
- Every command carries a
.tag, and only sentences that return the same tag are kept for that key. - A timeout, a reset or a
!fatalsets that key to empty, reconnects, and carries on, at most three times. - Heavy tables run last, with timeouts of 60 to 90 seconds, inside an overall budget that stops new commands before the caller's hard kill.
- Per-command outcomes go to a separate status file, so the dumps that other tools read stay clean.
The general lesson is older than this API. If a protocol gives you a correlation ID, use it on every message, even when the happy path never needs it. The unhappy path arrives late, which is the whole point.