Problems with netgraph

[Oleksandr Samoylyk]

Julian Elischer wrote:
> Oleksandr Samoylyk wrote:
>> Dear developers,
>>
>> Please read this thread:
>> http://lists.freebsd.org/pipermail/freebsd-net/2008-May/017891.html
>>
>> I'm using no encryption and no compression in mpd, so netgraph should 
>> fly.
>> It seems to get bad after 500 sessions...
>>
>> In FreeBSD 6 - it's swi1: net - 100% CPU
>> In FreeBSD 7 - it's em0 taskq - 100% CPU
> 
> unfortunatly I've been totally  ignoring this thread because it said
> "trouble with em" in the topic..
> If you'd said
> trouble with mpd  then maybe I'd have looked earlier..
> 
>>
>> After playing with it I can make guess that's a netgraph problem.
>>
>> Maybe it's a poor design in netgraph the way it handle its tables and 
>> it's is probably not designed to add so many interfaces.
> 
> how many?
> 

# vmstat -m | grep netgraph
  netgraph_msg     0     0K       -  1355522  64,128,256,512,1024
netgraph_node 12110  3028K       -    28829  256
netgraph_hook 29648  3706K       -    66104  128
      netgraph  8076 12663K       -    15209  64,256,1024,4096
netgraph_sock     4     1K       -     5489  128
netgraph_path     1     1K       -   728112  16,32
netgraph_mppc     0     0K       -        1  1024
netgraph_ksock  1349   169K       -     3414  128
netgraph_iface  1346   169K       -     2367  128
  netgraph_ppp  1346 16152K       -     2367
  netgraph_bpf 16020  2003K       -    33296  128

# ifconfig -a | grep ng | wc -l
     1341

I saw a peak with about 1900 ng interfaces. That was tooo sloowwww....

last pid: 39304;  load averages:  1.12,  1.23,  1.20 
                                            up 0+01:15:38  22:39:04
93 processes:  10 running, 70 sleeping, 13 waiting
CPU states:  0.3% user,  0.0% nice, 15.3% system,  3.1% interrupt, 81.3% 
idle
Mem: 46M Active, 7772K Inact, 109M Wired, 128K Cache, 16M Buf, 7756M Free
Swap: 4096M Total, 4096M Free

   PID USERNAME  THR PRI NICE   SIZE    RES STATE  C   TIME   WCPU COMMAND
   819 root       23  50    0 92764K 36424K select 1   0:00 207.18% mpd5
    17 root        1 171 ki31     0K    16K RUN    1  71:41 100.00% 
idle: cpu1
    14 root        1 171 ki31     0K    16K CPU4   4  69:16 100.00% 
idle: cpu4
    15 root        1 171 ki31     0K    16K CPU3   3  68:11 100.00% 
idle: cpu3
    29 root        1 -68    -     0K    16K CPU6   6  60:50 100.00% em0 
taskq
    18 root        1 171 ki31     0K    16K CPU0   0  71:58 97.17% idle: 
cpu0
    16 root        1 171 ki31     0K    16K CPU2   2  68:13 95.65% idle: 
cpu2
    11 root        1 171 ki31     0K    16K CPU7   7  66:23 86.38% idle: 
cpu7
    13 root        1 171 ki31     0K    16K CPU5   5  65:28 71.97% idle: 
cpu5
    19 root        1 -44    -     0K    16K WAIT   5  19:53 31.59% swi1: net
    30 root        1 -68    -     0K    16K -      7   8:41 12.89% em1 taskq
    12 root        1 171 ki31     0K    16K RUN    6  12:41  0.00% idle: 
cpu6
    20 root        1 -32    -     0K    16K WAIT   3   1:12  0.00% swi4: 
clock

# uname -v
FreeBSD 7.0-STABLE #0: Mon May  5 01:11:23 EEST 2008


CPU: Intel(R) Xeon(R) CPU           E5335  @ 2.00GHz (2000.01-MHz 
K8-class CPU)
   Origin = "GenuineIntel"  Id = 0x6f7  Stepping = 7
 
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
   Features2=0x4e33d<SSE3,RSVD2,MON,DS_CPL,VMX,TM2,SSSE3,CX16,xTPR,PDCM,DCA>
   AMD Features=0x20100800<SYSCALL,NX,LM>
   AMD Features2=0x1<LAHF>
   Cores per package: 4

FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs

usable memory = 8580038656 (8182 MB)
avail memory  = 8292810752 (7908 MB)

# ngctl list
ngctl: send msg: No buffer space available

# cat /boot/loader.conf
loader_logo="beastie"
autoboot_delay="3"
hw.em.rxd="1024"
hw.em.txd="1024"
net.inet.tcp.tcbhashsize="4096"
kern.ipc.maxpipekva="32000000"
net.graph.recvspace="128000"
net.graph.maxdgram="128000"

Linux with poptop was running quit smoothly. I thought that netgraph 
would be fast as hell.

-- 
  Oleksandr Samoylyk
  OVS-RIPE