IRQ31 and IRQ32 on HPDL585 running FreeBSD 7.0 are consuming HIGH CPU usage

Fri Nov 14 03:43:51 PST 2008

> ----- Original Message ----

> From: Won De Erick <won.derick at yahoo.com>
> To: Jeremy Chadwick <koitsu at FreeBSD.org>
> Cc: freebsd-hardware at freebsd.org
> Sent: Thursday, November 13, 2008 4:07:37 PM
> Subject: Re: IRQ31 and IRQ32 on HPDL585 running FreeBSD 7.0 are consuming HIGH CPU usage
> 
> Noted on this, I will update you through this thread.

With FreeBSD 7.1 Beta2, here is the result:

  52 root        1 -68    -     0K    16K CPU11  b  38:43 95.36% irq32: bce1
  51 root        1 -68    -     0K    16K CPU10  a  25:50 85.16% irq31: bce0

There's a slight difference w/ the previous result though, but I observed that overall CPU utilization didn't change.

### Complete result:

  PID USERNAME  THR PRI NICE   SIZE    RES STATE  C   TIME   WCPU COMMAND
   21 root        1 171 ki31     0K    16K CPU5   5  91:29 100.00% idle: cpu5
   14 root        1 171 ki31     0K    16K CPU12  c  91:29 100.00% idle: cpu12
   22 root        1 171 ki31     0K    16K CPU4   4  91:29 100.00% idle: cpu4
   17 root        1 171 ki31     0K    16K CPU9   9  91:28 100.00% idle: cpu9
   12 root        1 171 ki31     0K    16K CPU14  e  91:27 100.00% idle: cpu14
   23 root        1 171 ki31     0K    16K CPU3   3  90:58 100.00% idle: cpu3
   25 root        1 171 ki31     0K    16K CPU1   1  86:41 100.00% idle: cpu1
   19 root        1 171 ki31     0K    16K CPU7   7  84:00 100.00% idle: cpu7
   24 root        1 171 ki31     0K    16K CPU2   2  83:53 100.00% idle: cpu2
   18 root        1 171 ki31     0K    16K CPU8   8  79:01 100.00% idle: cpu8
   26 root        1 171 ki31     0K    16K RUN    0  74:18 100.00% idle: cpu0
   11 root        1 171 ki31     0K    16K CPU15  0   0:00 100.00% idle: cpu15
   13 root        1 171 ki31     0K    16K CPU13  0   0:00 100.00% idle: cpu13
   20 root        1 171 ki31     0K    16K CPU6   6  90:18 99.27% idle: cpu6
   52 root        1 -68    -     0K    16K CPU11  b  38:43 95.36% irq32: bce1
   51 root        1 -68    -     0K    16K CPU10  a  25:50 85.16% irq31: bce0
   16 root        1 171 ki31     0K    16K RUN    a  65:39 15.97% idle: cpu10
   28 root        1 -32    -     0K    16K WAIT   8  12:28  5.18% swi4: clock sio
   15 root        1 171 ki31     0K    16K RUN    b  52:46  3.76% idle: cpu11
   45 root        1 -64    -     0K    16K WAIT   7   7:29  1.17% irq17: uhci0
   47 root        1 -64    -     0K    16K WAIT   6   1:11  0.10% irq16: ciss0

Is there some ways how the intensive [network] load can be distributed among the IDLE processors?

Another thing, I observed that in the above test, the net.isr is enabled by default. When I tried disabling this,

# sysctl net.isr.direct=0
net.isr.direct: 1 -> 0

the result:

52 root        1 -68    -     0K    16K WAIT   b  64:00 42.97% irq32: bce1
51 root        1 -68    -     0K    16K WAIT   a  38:22 12.26% irq31: bce0

The CPU utilizations considerably dropped!
What was changed in the fbsd7.1?
How useful this net.isr is? Is this needed in an environment with heavy network traffic? Can someone explain?

> However is there any possibility of the following:
> 
> > I don't know if there's a way to split the interrupt request for each bce's Rx and Tx,
> > which means a total of four IRQs, and eventually four cores (or 4 CPUs)
> > for the transactions. With this way, the IDLE processors would be utilized.
> 
> What I mean here is, for the two interfaces:
> 
> one IRQ for bce0 Rx
> one IRQ for bce0 Tx
> one IRQ for bce1 Rx
> one IRQ for bce1 Tx
> 
> 
> Thanks,
> 
> Won
> 
> 
> 
> ________________________________
> From: Jeremy Chadwick <koitsu at FreeBSD.org>
> To: Won De Erick <won.derick at yahoo.com>
> Cc: freebsd-hardware at freebsd.org
> Sent: Thursday, November 13, 2008 3:46:02 PM
> Subject: Re: IRQ31 and IRQ32 on HPDL585 running FreeBSD 7.0 are consuming HIGH CPU usage
> 
> On Wed, Nov 12, 2008 at 11:38:15PM -0800, Won De Erick wrote:
> > I am conducting a CPU utilization testing with my box(HP DL 585 running FreeBSD 7.0), and come up with the results below:
> > 
> > 52 root        1 -68    -     0K    16K CPU11  b 123:53 100.00% irq32: bce1
> > 51 root        1 -68    -     0K    16K CPU10  a 119:28 89.06% irq31: bce0
> > 
> > irq31 and irq32 are consuming high CPU usage, which i think the cause of hard reset.
> 
> There was a ***major*** bce(4) cleanup that just happened.  Your 7.0 box
> will not have these changes.  Please upgrade your box to RELENG_7
> (a.k.a. 7.1-PRERELEASE), csup'd recently (today preferably), and try
> your tests again:
> 
> http://lists.freebsd.org/pipermail/freebsd-stable/2008-November/046482.html
> 
> -- 
> | Jeremy Chadwick                                jdc at parodius.com |
> | Parodius Networking                      http://www.parodius.com/ |
> | UNIX Systems Administrator                  Mountain View, CA, USA |
> | Making life hard for others since 1977.              PGP: 4BD6C0CB |
> 