Determining cause of transfer limit

[Sean C. Farley]

On Sat, 21 Nov 2020, Sean C. Farley wrote:

> I have recently upped my Internet service and have now noticed a limit being 
> reached, but I am not certain which limit and best option to resolve it.
>
> I am using a circa 2007 system as a multi-purpose router running FreeBSD 
> 12-STABLE (r367740).  The issue is that it maxes out around 400Mb/s when 
> running a speed test through it between my workstation and various test sites 
> (i.e., DSL Reports and Speedtest).  There are two NIC's (both are Intel 
> 82541PI) in use with one to the ISP and one to my workstation.
>
> At first, I saw one of them apparently hitting an interrupt rate of just over 
> 8000, so I bumped their rate limits higher with little to no improvement.
>
> What makes me believe I can theoretically get faster speeds is that I can use 
> the onboard NIC (Marvell 88E8056) to replace one of the NIC's and nearly 
> double the speed.  The difference is that it is on the PCI-E bus and has MSI 
> support.
>
> irq16: em0:irq0+
> irq17: em1:irq0
> irq20: hpet0
> irq258: mskc0
>
> I have many network settings, but changing them did nothing.  Here are the 
> settings I am trying now that seem to squeak a little extra performance.  The 
> commented-out lines are ones I tried without seeing any change.  I have also 
> tested without these settings.
>
> /boot/loader.conf
> hw.em.rx_process_limit="-1"
> # dev.em.0.iflib.override_nrxds="2048"
> # dev.em.1.iflib.override_nrxds="2048"
> # dev.em.2.iflib.override_nrxds="2048"
> # dev.em.0.iflib.override_ntxds="2048"
> # net.link.ifqmaxlen="2048"
> hw.em.max_interrupt_rate="32000"
> # net.isr.maxthreads="-1"
> # net.isr.bindthreads="1"
>
> /etc/sysctl.conf
> kern.random.harvest.mask=351
> dev.em.0.fc=0
> dev.em.1.fc=0
> dev.em.0.itr=122        # Allow past 8000 interrupts/second.
> dev.em.1.itr=122
> net.inet.ip.redirect=0
> net.inet6.ip6.redirect=0
>
> Increasing these from 66 to 250 did not help:
> hw.em.rx_abs_int_delay: 66
> hw.em.tx_abs_int_delay: 66
> hw.em.tx_int_delay: 66
>
> I am utilizing pf, but I doubt it is the issue since using the same rules 
> with the msk driver would have held the speed down to 400Mb/s.
>
> Am I hitting the limit of the PCI bus (memory or interrupt) or something 
> else?  I can buy a new PCI-E NIC for the internal network, but I rather fully 
> utilize the Intel NIC's I have, if possible.

Well, I surrendered with using those NIC's.  If anyone runs across this 
thread in the future, I want to let you know that I fixed my issue by 
buying a used HP multi-port NIC using the Intel 82580 chipset.  The only 
setting I made was to turn off flowcontrol.  I may experiment with 
tweaks (such as bumping dev.igb.N.iflib.override_nrxds) in the future, 
but it looks like I am in a much better situation already.

Sean
-- 
scf at FreeBSD.org