[fbsd] Re: [fbsd] Network performance in a dual CPU system

[Jeremie Le Hen]

Hi, Robert,

On Thu, Apr 27, 2006 at 02:54:21PM +0100, Robert Watson wrote:
> 
> On Thu, 27 Apr 2006, Jeremie Le Hen wrote:
> 
> >>  PID USERNAME   THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
> >>   60 root         1 -44 -163     0K     8K WAIT   355.6H 72.17% swi1:
> >>net
> >>   39 root         1 -68 -187     0K     8K WAIT    52.3H  5.22% irq28:
> >>bge0
> >>   40 root         1 -68 -187     0K     8K WAIT    28.3H  2.25% irq29:
> >>bge1
> >>   11 root         1 171   52     0K     8K RUN    166.6H  0.00% idle
> >>   63 root         1 -16    0     0K     8K -      121:55  0.00% yarrow
> >>   61 root         1 -32 -151     0K     8K WAIT    46:21  0.00% swi4:
> >>clock sio
> >>[...]
> >>
> >>Does anyone know whether a dual CPU system can help us improve the 
> >>situation? I was wondering if the software interrupt threads would be 
> >>divided between the two processors.
> >
> >I am a few weeks late, I just saw this very interesting thread.  What 
> >solution did you finally employ to circumvent your high interrupt load ?
> 
> I missed the original thread, but in answer to the question: if you set 
> net.isr.direct=1, then FreeBSD 6.x will run the netisr code in the ithread 
> of the network device driver.  This will allow the IP forwarding and 
> related paths in two threads instead of one, potentially allowing greater 
> parallelism. Of course, you also potentially contend more locks, you may 
> increase the time it takes for the ithread to respond to new interrupts, 
> etc, so it's not quite cut and dry, but with a workload like the one shown 
> above, it might make quite a difference.

Actually you already replied in the original thread, explaining mostly
the same thing.  The whole thread [1] brought up multiple valuable
network performance tuning knobs, such as polling, fastforwarding,
net.isr.direct but there is no happy end to the thread.  Given this is
a real world situation, I wanted to know how Marcos revolved his
problem.

BTW, what I understand is that net.isr.direct=1 prevents from
multiplexing all packets on the netisr thread and instead makes the
ithread do the job.  In this case, what happens to the netisr thread ?
Does it still have some work to do or is it removed ?


Thank you.
Regards,

[1] http://lists.freebsd.org/pipermail/freebsd-net/2006-February/thread.html#9725
-- 
Jeremie Le Hen
< jeremie at le-hen dot org >< ttz at chchile dot org >