[fbsd] Network performance in a dual CPU system

[Robert Watson]

On Thu, 27 Apr 2006, Jeremie Le Hen wrote:

>>   PID USERNAME   THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
>>    60 root         1 -44 -163     0K     8K WAIT   355.6H 72.17% swi1:
>> net
>>    39 root         1 -68 -187     0K     8K WAIT    52.3H  5.22% irq28:
>> bge0
>>    40 root         1 -68 -187     0K     8K WAIT    28.3H  2.25% irq29:
>> bge1
>>    11 root         1 171   52     0K     8K RUN    166.6H  0.00% idle
>>    63 root         1 -16    0     0K     8K -      121:55  0.00% yarrow
>>    61 root         1 -32 -151     0K     8K WAIT    46:21  0.00% swi4:
>> clock sio
>> [...]
>>
>> Does anyone know whether a dual CPU system can help us improve the 
>> situation? I was wondering if the software interrupt threads would be 
>> divided between the two processors.
>
> I am a few weeks late, I just saw this very interesting thread.  What 
> solution did you finally employ to circumvent your high interrupt load ?

I missed the original thread, but in answer to the question: if you set 
net.isr.direct=1, then FreeBSD 6.x will run the netisr code in the ithread of 
the network device driver.  This will allow the IP forwarding and related 
paths in two threads instead of one, potentially allowing greater parallelism. 
Of course, you also potentially contend more locks, you may increase the time 
it takes for the ithread to respond to new interrupts, etc, so it's not quite 
cut and dry, but with a workload like the one shown above, it might make quite 
a difference.

Robert N M Watson