Interrupt routine usage not shown by top in 8.0

[Barney Cordoba]

--- On Thu, 3/12/09, Scott Long <scottl at samsco.org> wrote:

> From: Scott Long <scottl at samsco.org>
> Subject: Re: Interrupt routine usage not shown by top in 8.0
> To: barney_cordoba at yahoo.com
> Cc: current at freebsd.org
> Date: Thursday, March 12, 2009, 7:42 PM
> Barney Cordoba wrote:
> > I'm fireing 400Kpps at a udp blackhole port.
> I'm getting 6000 interrupts
> > per second on em3:
> > 
> > testbox# vmstat -i; sleep 1; vmstat -i
> > interrupt                          total       rate
> > irq1: atkbd0                           1          0
> > irq6: fdc0                             1          0
> > irq17: uhci1+                       2226          9
> > irq18: uhci2 ehci+                     9          0
> > cpu0: timer                       470507       1993
> > irq256: em0                          665          2
> > irq259: em3                      1027684       4354
> > cpu1: timer                       470272       1992
> > cpu3: timer                       470273       1992
> > cpu2: timer                       470273       1992
> > Total                            2911911      12338
> > 
> > interrupt                          total       rate
> > irq1: atkbd0                           1          0
> > irq6: fdc0                             1          0
> > irq17: uhci1+                       2226          9
> > irq18: uhci2 ehci+                     9          0
> > cpu0: timer                       472513       1993
> > irq256: em0                          668          2
> > irq259: em3                      1033703       4361
> > cpu1: timer                       472278       1992
> > cpu3: timer                       472279       1992
> > cpu2: timer                       472279       1992
> > Total                            2925957      12345
> > 
> > 
> > top -SH shows:
> > 
> >   PID  STATE  C   TIME    CPU COMMAND
> >    10  CPU3   3   7:32 100.00% idle
> >    10  CPU2   2   7:32 100.00% idle
> >    10  RUN    0   7:31 100.00% idle
> >    10  CPU1   1   7:31 100.00% idle
> > 
> > This implies that CPU usage is substantially
> under-reported in general
> > by the system. Note that I've modified
> em_irq_fast() to call em_handle_rxtx() directly rather than
> scheduling a task to illustrate
> > the problem
> > 
> 
> With unmodified code, what do you see?  Are you sending
> valid UDP frames with valid checksums and a valid port, or
> is everything that you're blasting at the interface
> getting dropped right away?  Calling em_handle_rxtx()
> directly will cause a very quick panic once you start
> handling real traffic and you encounter a lock.
> 
> Scott

I think you're mistaken. I'm also accessing the system via an em port
(and running top) and em_handle_rxtx() is self contained lock-wise. 
The taskqueue doesn't obtain a lock before calling the routine.

As I mentioned, they're being dumped into a udp blackhole, which implies
that I have udp.blackhole set and the port is unused. I can see the 
packets hit the udp socket so its working as expected:

853967872 dropped due to no socket

With unmodified code, the tasq shows 25% usage or so.

I'm not sure what the point of your criticism for what clearly is a test.
Are you implying that the system can receive 400K pps with 6000 ints/sec
and record 0% usage because of a coding imperfection? Or are you implying
that the 25% usage is all due to launching tasks unnecessarily and process 
switching?

Barney