unexplained latency, interrupt spikes and loss of throughput on FreeBSD router/firewall system

[mike tancsa]

On 1/15/2020 9:55 AM, John Jasen wrote:
> Executive summary:
>
> Periodically, load will spike on network interrupts on one of our
> firewalls. Latency will quickly climb to the point that things are
> unresponsive, sessions will timeout, and bandwidth will plummet.

A couple of wild stabs... Are the routers generating any odd amount of
ICMP response traffic at the time ? e.g. port|host unreachable etc ?
(maybe track netstat -s -p icmp). Are there any bursts of icmp redirects
happening ? I know that can slog a router sometimes-- Try instrumenting
the appropriate oids (sysctl -a | grep -i redirect)  to see if thats the
case.  A lot of small packets ?  If possible maybe a network tap in
front of the boxes to capture / profile the traffic before/after to see
if there is something like a big scan happening or DOS with many small
packets etc.  If thats not possible, do you have enough spare CPU to do
some netflow analysis on the box ? Or maybe take some periodic snapshots
of the interface stats and compare normal to bad periods via sysctl -A
dev.cxl | grep "_frames_"

Good luck!

    ---Mike