Problem with process parallelization
Zin'kov Oleg
noc at bg.net.ua
Mon Sep 29 07:41:43 UTC 2008
>> Hello, freebsd-net mailing list.
>>
>> We have server such configurtion:
>> - 2 quadcore AMD Opteron processors;
>> - 4 GB RAM;
>> - NIC Intel Pro/1000 PT, Dual Port Server Adapter.
>>
>> ###########################################################
>>
>> Problem:
>>
>> in some moments of time, at the growth of the network activity, one of
>> the processors is fully loaded at 100%.
>>
>> ###########################################################
>>
>> Kernel configuration:
>>
>> FreeBSD atlantis.bg.net.ua 7.0-STABLE FreeBSD 7.0-STABLE #1: Tue Apr 1
>> 15:06:30 EEST 2008
>> root at atlantis.bg.net.ua:/usr/obj/usr/src/sys/ATLANTIS amd64
>>
>> /etc/sysctl.conf:
>>
>> net.inet.tcp.blackhole=2
>> net.inet.udp.blackhole=1
>> kern.ipc.somaxconn=16384
>> net.inet.ip.fastforwarding=1
>> net.inet.ip.maxfragpackets=2000
>> net.inet.ip.intr_queue_maxlen=1000
>> net.inet.ip.dummynet.hash_size=2048
>> net.inet.tcp.recvspace=65536
>> net.inet.udp.recvspace=65536
>> net.inet.raw.recvspace=32768
>> net.local.stream.recvspace=32768
>> net.local.dgram.recvspace=32768
>> net.local.stream.sendspace=32768
>> net.inet.tcp.sendspace=65536
>> net.inet.icmp.icmplim=500
>> dev.em.0.rx_int_delay=500
>> dev.em.0.tx_int_delay=500
>> dev.em.0.rx_abs_int_delay=800
>> dev.em.0.tx_abs_int_delay=800
>> dev.em.1.rx_int_delay=500
>> dev.em.1.tx_int_delay=500
>> dev.em.1.rx_abs_int_delay=800
>> dev.em.1.tx_abs_int_delay=800
>> net.link.ether.inet.max_age=600
>>
>> /boot/loader.conf:
>>
>> hw.em.rxd=4096
>> hw.em.txd=4096
>>
>> /etc/rc.firewall:
>>
>> 82 pipes like theese:
>>
>> pipe 387 ip from any to 193.227.x.x in recv vlan10
>> pipe 388 ip from 193.227.x.x to any out xmit vlan10
>>
>>
>> #########################################
>> Kernel:
>>
>>
>> cpu HAMMER
>> ident ATLANTIS
>>
>> # To statically compile in device wiring instead of /boot/device.hints
>> #hints "GENERIC.hints" # Default places to look for
>> devices.
>>
>> makeoptions DEBUG=-g # Build kernel with gdb(1) debug
>> symbols
>>
>> options SCHED_ULE # 4BSD scheduler
>> options PREEMPTION # Enable kernel thread
>> preemption
>> options INET # InterNETworking
>> #options SCTP # Stream Control Transmission
>> Protocol
>> options FFS # Berkeley Fast Filesystem
>> options
>> SOFTUPDATES # Enable FFS soft updates support
>> options
>> UFS_ACL # Support for access control lists
>> options
>> UFS_DIRHASH # Improve performance on big directories
>> options PROCFS # Process filesystem (requires
>> PSEUDOFS)
>> options PSEUDOFS # Pseudo-filesystem framework
>> options GEOM_PART_GPT # GUID Partition Tables.
>> options GEOM_LABEL # Provides labelization
>> options COMPAT_43TTY # BSD 4.3 TTY compat [KEEP
>> THIS!]
>> options COMPAT_IA32 # Compatible with i386 binaries
>> options COMPAT_FREEBSD4 # Compatible with FreeBSD4
>> options
>> COMPAT_FREEBSD5 # Compatible with FreeBSD5 options
>> COMPAT_FREEBSD6 # Compatible with FreeBSD6 options
>> KTRACE
>> # ktrace(1) support
>> options SYSVSHM # SYSV-style shared memory
>> options
>> SYSVMSG # SYSV-style message queues options
>> SYSVSEM # SYSV-style semaphores
>> options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time
>> extensions
>> options KBD_INSTALL_CDEV # install a CDEV entry in /dev
>> options ADAPTIVE_GIANT # Giant mutex is adaptive.
>> options
>> STOP_NMI # Stop CPUS using NMI instead of IPI
>> options AUDIT # Security event auditing
>>
>> # Make an SMP-capable kernel by default
>> options SMP # Symmetric MultiProcessor
>> Kernel
>>
>> # Bus support.
>> device acpi
>> device pci
>>
>> # ATA and ATAPI devices
>> device ata
>>
>> device atadisk # ATA disk drives
>> options ATA_STATIC_ID # Static device numbering
>>
>> # RAID controllers
>> device twe # 3ware ATA RAID
>>
>> # atkbdc0 controls both the keyboard and the PS/2 mouse
>> device atkbdc # AT keyboard controller
>> device atkbd # AT keyboard
>>
>> device vga # VGA video card driver
>>
>> device splash # Splash screen and screen saver support
>>
>> # syscons is the default console driver, resembling an SCO console
>> device
>> sc
>>
>> ### COM
>> device sio
>>
>> # PCI Ethernet NICs.
>> device em # Intel PRO/1000 adapter Gigabit
>> Ethernet
>> Card
>>
>> # PCI Ethernet NICs that use the common MII bus controller code.
>> # NOTE: Be sure to keep the 'device miibus' line in order to use these
>> NICs! device miibus # MII bus support
>> device bge # Broadcom BCM570xx Gigabit Ethernet
>> device fxp # Intel EtherExpress PRO/100B (82557,
>> 82558)
>>
>> # Pseudo devices.
>> device loop # Network loopback
>> device random # Entropy device
>> device ether # Ethernet support
>> device pty # Pseudo-ttys (telnet etc)
>> device vlan
>>
>> # The `bpf' device enables the Berkeley Packet Filter.
>> # Be aware of the administrative consequences of enabling this!
>> # Note that 'bpf' is required for DHCP.
>> device bpf # Berkeley packet filter
>>
>> ## Custom options
>> # NetGraph
>> options NETGRAPH
>> options NETGRAPH_ONE2MANY
>> options NETGRAPH_NETFLOW
>> options NETGRAPH_CISCO
>> options NETGRAPH_ETHER
>> options NETGRAPH_KSOCKET
>> options NETGRAPH_SOCKET
>> options NETGRAPH_TEE
>>
>> options IPFIREWALL
>> options IPFIREWALL_VERBOSE
>> options IPFIREWALL_FORWARD
>> options IPFIREWALL_VERBOSE_LIMIT=1000
>> options IPFIREWALL_DEFAULT_TO_ACCEPT
>> options DUMMYNET
>> options HZ=1000
>> options DEVICE_POLLING
>> #####################################################
>>
>> Interfaces:
>> - em0
>> - em1
>> - bge0
>> - bge1
>> - vlan (61 virtual interfaces)
>>
>> #####################################################
>> top -S
>>
>> last pid: 9673; load averages: 1.94, 1.75, 1.57
>> up 0+19:17:21
>> 19:45:01
>> 77 processes: 11 running, 49 sleeping, 17 waiting
>> CPU states: 0.0% user, 0.0% nice, 22.6% system, 0.3% interrupt, 77.0%
>> idle Mem: 198M Active, 410M Inact, 455M Wired, 228K Cache, 214M Buf,
>> 2874M
>> Free Swap: 4096M Total, 4096M Free
>>
>> PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU
>> COMMAND
>> 11 root 1 171 ki31 0K 16K CPU7 7 19.0H 100.00% idle:
>> cpu7
>> 16 root 1 171 ki31 0K 16K CPU2 2 18.9H 100.00% idle:
>> cpu2
>> 17 root 1 171 ki31 0K 16K RUN 1 18.8H 100.00% idle:
>> cpu1
>> 13 root 1 171 ki31 0K 16K CPU5 5 18.8H 100.00% idle:
>> cpu5
>> 18 root 1 171 ki31 0K 16K CPU0 0 916:13 100.00% idle:
>> cpu0
>> 12 root 1 171 ki31 0K 16K CPU6 6 18.8H 99.85% idle:
>> cpu6
>> 35 root 1 -68 - 0K 16K CPU4 4 466:17 96.00% em1
>> taskq
>> 34 root 1 -68 - 0K 16K CPU3 3 482:01 90.38% em0
>> taskq
>> 15 root 1 171 ki31 0K 16K RUN 3 655:20 13.38% idle:
>> cpu3
>> 14 root 1 171 ki31 0K 16K RUN 4 671:52 3.08% idle:
>> cpu4
>>
>>
>> ##############################################
>> 19:45[p0]root at atlantis#~>netstat -w 1 -I em0
>> input (em0) output
>> packets errs bytes packets errs bytes colls
>> 57381 0 36442155 68726 0 69126050 0
>> 56817 0 37480502 67656 0 66053093 0
>> 57847 0 39532712 68603 0 67037042 0
>> 56908 0 37197022 68924 0 68660108 0
>> 57107 0 37643382 68398 0 68113937 0
>> 56847 0 35944754 68394 0 67896267 0
>> 58754 0 39763361 68966 0 70029090 0
>> 58343 0 38301796 69635 0 69948678 0
>> ^C
>> 19:46[p0]root at atlantis#~>netstat -w 1 -I em1
>> input (em1) output
>> packets errs bytes packets errs bytes colls
>> 67944 0 68877031 55376 0 36252905 0
>> 65943 0 66722222 54575 0 37710643 0
>> 64639 0 67149621 53298 0 35423539 0
>> 63988 0 65035759 51787 0 35402337 0
>> 63849 0 65968513 50727 0 31683425 0
>> 64301 0 66684912 50193 0 30917339 0
>>
>>
>>
>> ###################################################################
>>
>>
>> How can we solve this problem and parallelize em1:taskq kernel processes
>> between all 8 processors?
>
> # sysctl net.isr.direct=0
> would add one more kernel thread to handle your network traffic.
>
> Regards, Dmitriy.
>
A problem remained :(
>>
>>
>> --
>> ISP BGNet
>> 288-03-53
>> 246-68-98
>>
>> Zin'kov Oleg
>> System administrator
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> --
>> ISP BGNet
>> 288-03-53
>> 246-68-98
>>
>> Zin'kov Oleg
>> System administrator
>>
>> _______________________________________________
>> freebsd-net at freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>>
>
>
--
ISP BGNet
288-03-53
246-68-98
Zin'kov Oleg
System administrator
More information about the freebsd-net
mailing list