A PoP router has a carrier handover on one SFP and two fibre links to other sites on two more. The handover card on the dashboard read a few kilobits per second. The fibre links were busy. The natural reading was that the carrier egress was mislabelled, or that traffic had been re-routed.
We did the sum instead. The router terminates 126 client service tunnels. Over 55 days of uptime, the tunnels had transmitted 47.4 TB toward the clients. The carrier SFP had received 46.5 TB from the internet. In the other direction, 5.5 TB and 5.5 TB. Bytes that go to clients have to come from somewhere, and they came from that port. The carrier egress was exactly what it said it was.
Why the two readings
- The busy fibre links are the routed underlay. The same client flows cross them one layer down, encapsulated, on their way to the CPEs. That is not a competing egress; it is the same traffic twice.
- The "idle" reading was a collector artefact: a byte delta taken over a degraded, short window. With the collector healthy, the card showed the port busy.
- A Layer 2 handover bridge reads close to zero bytes by design on some platforms, because the bridge itself is not where the packets are counted.
No ticket to the carrier. One fix to the monitoring. Before you report a link, add up the bytes on both sides of the box.