High intr CPU % and slow throughput

[Alexander V. Chernikov]

On 18.09.2014 20:26, Freddie Cash wrote:
> [Not sure if this is more appropriate for the -ipfw or -stable mailing
> lists.]
> 
> 
> 64-bit FreeBSD 10.0-p7
> 
> Dual-core AMD Opteron 1218 CPU @ 2.6 GHz
> ​2 GB of DDR2 RAM
> Intel i350-T4 quad-port gigabit NIC using igb(4)
> 
> Each of the gigabit NIC ports are connected to gigabit links (we have a
> gigabit fibre link to our ISP, which has dual 10 Gbps links to the public
> Internet).
> 
> Using the following simple ruleset (there are more rules, but these are the
> ones that match when we test transfers to/from the Internet):
Please show all the ruleset with counters.

> 
> ipfw nat 8668 config ip 142.24.
> ​x.y​
> same_ports
> 
> 10 allow ip from any to any via lo0
> 12 allow carp from any to any
> 
> 20 reject log logamount 10000 ip from 10.0.0.0/8 to any in recv igb0
> 22 reject log logamount 10000 ip from 127.0.0.0/8 to any in recv igb0
> ​2​
> 4 reject log logamount 10000 ip from 172.16.0.0/20 to any in recv igb0
> 26 reject log logamount 10000 ip from 192.168.0.0/16 to any in recv igb0
> 
> 50 skipto 65000 ip from 192.168.0.0/24 to not 142.24.
> ​x.z
> /25 in recv igb2
> 52 skipto 65000 ip from not 142.24.13.128/25 to 142.24.
> ​x.y
>  in recv igb0
> 
> 65000 allow ip from 192.168.0.0/24 to any in recv igb2
> 65002 nat 8668 ip from 192.168.0.0/24 to any out xmit igb0
> 65004 allow ip from 142.24.
> ​x.y​
> to any out xmit igb0
> 
> 65006 nat 8668 ip from any to 142.24.
> ​x.y​
> in recv igb0
> 65008 allow ip from any to 192.168.0.0/24 in recv igb0
> 65010 allow ip from any to 192.168.0.0/24 out xmit igb2

> 
> When we start a large download or file transfer from the Internet (a single
> file from a single server), CPU usage for the [intr{irq256: igb0:que}]
> kernel thread jumps to over 90% (one CPU core) and causes all traffic
> through the firewall (even traffic that doesn't go through igb0) to grind
> to a standstill.  Some TCP connections through other interfaces are even
> dropped.​  During this time, the other CPU core is under 50% usage.
can you do the following:
kldload hwpmc
sudo pmcstat -TS instructions -w 1

and show its output when the problem is observed?
> 
> IIUIC, the [intr{irq256: igb0:que}] isn't showing actual CPU usage for
> processing hardware interrupts, but is showing the CPU usage used to
> process the packets going through IPFW.  Correct?  "vmstat -i" shows only
> 10-15 interrupts per second for each of the igb interfaces.
> 
> The really depressing part is that throughput (as shown by "iftop -i igb0"
> and snmp graphing) never goes above 40 Mbps.  :(
> 
> What can I do to try and track down exactly why this is occurring?
> 
> Is there anything I can do to reduce or mitigate this CPU usage?
> 
> Or, is this simply a case of the CPU being too old?
> 
> /boot/loader.conf currently has the following (been playing with most of
> these lately, without much change in CPU usage):
> 
> ##  Tune the igb(4) interfaces a little
> hw.igb.enable_aim="1"
> hw.igb.enable_msix="1"
> hw.igb.header_split="0"
> hw.igb.max_interrupt_rate="16000"
> hw.igb.num_queues="0"
> hw.igb.rx_process_limit="1000"
> hw.igb.rxd="4096"
> hw.igb.txd="4096"
> 
> ##  Configure kernel
> kern.hz="4000"
> 
> ##  Configure IPFW
> net.inet.ip.fw.default_to_accept="1"
> net.inet.ip.fw.verbose="1"
> 
> ##  Configure network threads
> net.isr.bindthreads="1"
> net.isr.direct="1"
> net.isr.maxthreads="2"
> 
> 
> ​/etc/sysctl.conf has the following (haven't changed these in a long time):
> 
> ​# IPFW options
> net.inet.ip.fw.autoinc_step=2
> net.inet.ip.fw.enable=1
> net.inet.ip.fw.one_pass=1
> net.inet.ip.fw.verbose=1
> net.inet.ip.fw.verbose_limit=10000
> 
> 
> At lunch today, we'll be failing-over to the other firewall, which will be
> running without any /boot/loader.conf or /etc/sysctl.conf entries to see if
> my "optimisations" are actually "pessimisations".
> 
>