Determining cause of transfer limit

[Sean C. Farley]

I have recently upped my Internet service and have now noticed a limit 
being reached, but I am not certain which limit and best option to 
resolve it.

I am using a circa 2007 system as a multi-purpose router running FreeBSD 
12-STABLE (r367740).  The issue is that it maxes out around 400Mb/s when 
running a speed test through it between my workstation and various test 
sites (i.e., DSL Reports and Speedtest).  There are two NIC's (both are 
Intel 82541PI) in use with one to the ISP and one to my workstation.

At first, I saw one of them apparently hitting an interrupt rate of just 
over 8000, so I bumped their rate limits higher with little to no 
improvement.

What makes me believe I can theoretically get faster speeds is that I 
can use the onboard NIC (Marvell 88E8056) to replace one of the NIC's 
and nearly double the speed.  The difference is that it is on the PCI-E 
bus and has MSI support.

irq16: em0:irq0+
irq17: em1:irq0
irq20: hpet0
irq258: mskc0

I have many network settings, but changing them did nothing.  Here are 
the settings I am trying now that seem to squeak a little extra 
performance.  The commented-out lines are ones I tried without seeing 
any change.  I have also tested without these settings.

/boot/loader.conf
hw.em.rx_process_limit="-1"
# dev.em.0.iflib.override_nrxds="2048"
# dev.em.1.iflib.override_nrxds="2048"
# dev.em.2.iflib.override_nrxds="2048"
# dev.em.0.iflib.override_ntxds="2048"
# net.link.ifqmaxlen="2048"
hw.em.max_interrupt_rate="32000"
# net.isr.maxthreads="-1"
# net.isr.bindthreads="1"

/etc/sysctl.conf
kern.random.harvest.mask=351
dev.em.0.fc=0
dev.em.1.fc=0
dev.em.0.itr=122        # Allow past 8000 interrupts/second.
dev.em.1.itr=122
net.inet.ip.redirect=0
net.inet6.ip6.redirect=0

Increasing these from 66 to 250 did not help:
hw.em.rx_abs_int_delay: 66
hw.em.tx_abs_int_delay: 66
hw.em.tx_int_delay: 66

I am utilizing pf, but I doubt it is the issue since using the same 
rules with the msk driver would have held the speed down to 400Mb/s.

Am I hitting the limit of the PCI bus (memory or interrupt) or something 
else?  I can buy a new PCI-E NIC for the internal network, but I rather 
fully utilize the Intel NIC's I have, if possible.

Sean
-- 
scf at FreeBSD.org