increasing em(4) buffer sizes

[rihad]

On 05/20/2010 05:19 PM, Eugene Grosbein wrote:
> On 20.05.2010 13:06, rihad wrote:
>
>>>>> Some more useful tunnables for loader.conf:
>>>>>
>>>>> dev.em.0.rx_int_delay=200
>>>>> dev.em.0.tx_int_delay=200
>>>>> dev.em.0.rx_abs_int_delay=200
>>>>> dev.em.0.tx_abs_int_delay=200
>>>>> dev.em.0.rx_processing_limit=-1
>>>>>
>>>> So this interrupt delay is the much talked about interrupt moderation?
>>>> Thanks, I'll try them. Is there any risk the machine won't boot with
>>>> them if rebooted remotely?
>>>
>>> Yes, for hw.em.rxd/hw.em.txd only. em interfaces will not function
>>> if you set these two too high for your nic chip, so read man em
>>> and mentioned Intel document carefully.
>>>
>>> It's pretty safe to play with other tunnables.
>>>
>> Can I even set the mentioned dev.em.0.* settings on a live system
>> remotely? You seem to also have an em card, have you tried doing that?
>
> Yes, it is permitted.
>
>

I haven't tweaked rx_abs_int_delay yet, but I've read the Intel specs 
that you suggested. It says in the paragraph 4.4:
  "The expected network use. High use implies high traffic rates, which 
makes the GbE controller more susceptible to overrun conditions if it 
delays interrupts too long."

Doesn't this mean that increasing the absolute timer (rx_abs_int_delay) 
would only aggravate the issue and the Ierrs will only increase? It's 
currently 66, by default I think:

dev.em.0.rx_int_delay: 0
dev.em.0.tx_int_delay: 66
dev.em.0.rx_abs_int_delay: 66
dev.em.0.tx_abs_int_delay: 66

I'm pretty sure interrupt overload is not an issue on our system:

Here's what top -HS currently shows:
    12 root     171 ki31     0K    16K CPU2   2 174.3H 100.00% idle: cpu2
    14 root     171 ki31     0K    16K CPU0   0 168.9H 99.76% idle: cpu0
    11 root     171 ki31     0K    16K RUN    3 150.4H 94.19% idle: cpu3
    13 root     171 ki31     0K    16K RUN    1 118.1H 73.97% idle: cpu1
    27 root     -68    -     0K    16K -      1  70.1H 31.88% em0 taskq
    42 root     -68    -     0K    16K -      3  30.5H  7.57% dummynet


The interrupt load is only 31% for em0.


So, hw.em.rxd=4096 seems to be the only way out? Increasing the buffer size.