Determining cause of transfer limit

[Sean C. Farley]

On Sat, 21 Nov 2020, Michael Sierchio wrote:

> Sorry for the top post.  Have you tried device polling? From
> /usr/src/sys/amd64/conf/NOTES:
...
> options         DEVICE_POLLING

*snip*

I had not tried it and had actually forgotten about that capability. 
Anyway, I just tried it (kernel built with polling and enabled via 
ifconfig on the devices) and saw no observable change.  I am not certain 
if the code paths can really make use of it with the switch to iflib.

However, I was able to squeak a bit more from the NIC's by the 
equivalent of polling(4) by bumping the delay for interrupts with the 
key one being hw.em.rx_int_delay.  It improved transfers going through 
the router, in both directions, by about 20Mb/s.  The number of 
interrupts dropped a lot, so I doubt I am hitting an interrupt limit.

/boot/loader.conf:
hw.em.rx_int_delay="264"
hw.em.rx_abs_int_delay="264"
hw.em.tx_int_delay="264"
hw.em.tx_abs_int_delay="264"

One other thing to note, is that when I am testing going through the 
router, there are many drops and stalls/restarts when downloading but 
not uploading.

Before test:
dev.em.1.iflib.txq0.r_abdications: 0
dev.em.1.iflib.txq0.r_restarts: 0
dev.em.1.iflib.txq0.r_stalls: 0
dev.em.1.iflib.txq0.r_starts: 9239
dev.em.1.iflib.txq0.r_drops: 0
dev.em.1.iflib.txq0.r_enqueues: 9240
dev.em.1.iflib.txq0.txq_cleaned: 9640
dev.em.1.iflib.txq0.txq_processed: 9680

After test:
dev.em.1.iflib.txq0.r_abdications: 14
dev.em.1.iflib.txq0.r_restarts: 7419
dev.em.1.iflib.txq0.r_stalls: 7419
dev.em.1.iflib.txq0.r_starts: 411677
dev.em.1.iflib.txq0.r_drops: 198
dev.em.1.iflib.txq0.r_enqueues: 878486
dev.em.1.iflib.txq0.txq_cleaned: 875005
dev.em.1.iflib.txq0.txq_processed: 875045

After 2nd test with Marvell as outgoing NIC starting from zero:
dev.em.1.iflib.txq0.r_abdications: 1
dev.em.1.iflib.txq0.r_restarts: 7647
dev.em.1.iflib.txq0.r_stalls: 7647
dev.em.1.iflib.txq0.r_starts: 1092018
dev.em.1.iflib.txq0.r_drops: 811
dev.em.1.iflib.txq0.r_enqueues: 1570425
dev.em.1.iflib.txq0.txq_cleaned: 1567629
dev.em.1.iflib.txq0.txq_processed: 1567671

Unless someone knows anything else, I think I am going to have to buy a 
PCI-E NIC for the router to replace one of the Intel NIC's.  I will 
either use it along with the Marvell or the other Intel NIC.  When 
testing between my workstation and any one Intel NIC on the router using 
iperf3, I can achieve just over 900Mb/s.

Sean
-- 
scf at FreeBSD.org



> On Sat, Nov 21, 2020 at 10:24 AM Sean C. Farley <scf at freebsd.org> wrote:
>
>> I have recently upped my Internet service and have now noticed a limit
>> being reached, but I am not certain which limit and best option to
>> resolve it.
>>
>> I am using a circa 2007 system as a multi-purpose router running FreeBSD
>> 12-STABLE (r367740).  The issue is that it maxes out around 400Mb/s when
>> running a speed test through it between my workstation and various test
>> sites (i.e., DSL Reports and Speedtest).  There are two NIC's (both are
>> Intel 82541PI) in use with one to the ISP and one to my workstation.
>>
>> At first, I saw one of them apparently hitting an interrupt rate of just
>> over 8000, so I bumped their rate limits higher with little to no
>> improvement.
>>
>> What makes me believe I can theoretically get faster speeds is that I
>> can use the onboard NIC (Marvell 88E8056) to replace one of the NIC's
>> and nearly double the speed.  The difference is that it is on the PCI-E
>> bus and has MSI support.
>>
>> irq16: em0:irq0+
>> irq17: em1:irq0
>> irq20: hpet0
>> irq258: mskc0
>>
>> I have many network settings, but changing them did nothing.  Here are
>> the settings I am trying now that seem to squeak a little extra
>> performance.  The commented-out lines are ones I tried without seeing
>> any change.  I have also tested without these settings.
>>
>> /boot/loader.conf
>> hw.em.rx_process_limit="-1"
>> # dev.em.0.iflib.override_nrxds="2048"
>> # dev.em.1.iflib.override_nrxds="2048"
>> # dev.em.2.iflib.override_nrxds="2048"
>> # dev.em.0.iflib.override_ntxds="2048"
>> # net.link.ifqmaxlen="2048"
>> hw.em.max_interrupt_rate="32000"
>> # net.isr.maxthreads="-1"
>> # net.isr.bindthreads="1"
>>
>> /etc/sysctl.conf
>> kern.random.harvest.mask=351
>> dev.em.0.fc=0
>> dev.em.1.fc=0
>> dev.em.0.itr=122        # Allow past 8000 interrupts/second.
>> dev.em.1.itr=122
>> net.inet.ip.redirect=0
>> net.inet6.ip6.redirect=0
>>
>> Increasing these from 66 to 250 did not help:
>> hw.em.rx_abs_int_delay: 66
>> hw.em.tx_abs_int_delay: 66
>> hw.em.tx_int_delay: 66
>>
>> I am utilizing pf, but I doubt it is the issue since using the same
>> rules with the msk driver would have held the speed down to 400Mb/s.
>>
>> Am I hitting the limit of the PCI bus (memory or interrupt) or something
>> else?  I can buy a new PCI-E NIC for the internal network, but I rather
>> fully utilize the Intel NIC's I have, if possible.
>>
>> Sean