Notes  / 

The WireGuard errors that weren't

Four to five transmit errors a second on a core router's WireGuard interface, flat regardless of load. They were keepalives to peers that no longer exist. The real packet loss was somewhere else entirely.

The interface counters on a core router showed a steady 4 to 5 transmit errors and drops per second on its WireGuard interface. That is the kind of number that gets a link blamed for everything. We checked whether it moved with load. It did not.

The router carries a service tunnel per client site, and 47 of those tunnels pointed at WireGuard peers that had been dead for a long time. Each dead peer gets a keepalive every 10 seconds. Forty-seven divided by ten is 4.7 a second. Measured: 4.1. The "errors" were the router trying to say hello to sites that left months ago. Cosmetic, and fixed by disabling the tunnels and peers for sites that are gone.

The real loss

The actual packet loss was in the queue tree. Each client service has a leaf queue shaped to its plan, with a default queue type that holds ten packets. Over eleven days of uptime the download leaves had dropped 15.6% of their packets, with nine services above 20%. The upload leaves dropped 2.9%, which matched the carrier handover port's transmit-queue drop counter exactly, so that counter is the upload shaper.

None of this is visible from the monitoring system, because RouterOS does not export queue drops over SNMP. The loudest counter was benign, the quiet one was the problem, and the one that mattered was not being exported at all. Count drops where the queue is, not where the counter is.

WireGuard   RouterOS   Queues   · Terawiz engineering notes