Hello Alexander,
Friday, December 27, 2013, 1:25:28 PM, you wrote:
>> Recently upgraded router system from 9.0-RELEASE to 9.2-STABLE and
>> got 100% CPU utilisation on all cores with interrupts under the same
>> load that had about 25-30% CPU utilisation before. Of course that lead
AVC> Looks interesting.
AVC> Are you sure all other configs/data load are the same?
Yes, everything was the same. Later changed NIC from 4 igbs to 1 ix.
AVC> I'm particularly interested in changes in: number of NIC queues, their
AVC> bindings and firewall ruleset.
igb0: <Intel(R) PRO/1000 Network Connection version - 2.3.10> port 0x3020-0x303f mem 0xc6b20000-0xc6b3ffff,0xc6b44000-0xc6b47fff irq 40 at device 0.0 on pci1
igb0: Using MSIX interrupts with 5 vectors
igb0: Ethernet address: 00:15:17:b9:ef:dc
igb0: Bound queue 0 to cpu 0
igb0: Bound queue 1 to cpu 1
igb0: Bound queue 2 to cpu 2
igb0: Bound queue 3 to cpu 3
igb1: <Intel(R) PRO/1000 Network Connection version - 2.3.10> port 0x3000-0x301f mem 0xc6b00000-0xc6b1ffff,0xc6b40000-0xc6b43fff irq 28 at device 0.1 on pci1
igb1: Using MSIX interrupts with 5 vectors
igb1: Ethernet address: 00:15:17:b9:ef:dd
igb1: Bound queue 0 to cpu 4
igb1: Bound queue 1 to cpu 5
igb1: Bound queue 2 to cpu 6
igb1: Bound queue 3 to cpu 7
pcib2: <ACPI PCI-PCI bridge> irq 24 at device 3.0 on pci0
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> irq 26 at device 5.0 on pci0
pci3: <ACPI PCI bus> on pcib3
igb2: <Intel(R) PRO/1000 Network Connection version - 2.3.10> port 0x2020-0x203f mem 0xc6420000-0xc643ffff,0xc6000000-0xc63fffff,0xc64c4000-0xc64c7fff irq 26 at device 0.0 on pci
3
igb2: Using MSIX interrupts with 5 vectors
igb2: Ethernet address: 00:1b:21:4a:69:78
igb2: Bound queue 0 to cpu 8
igb2: Bound queue 1 to cpu 9
igb2: Bound queue 2 to cpu 10
igb2: Bound queue 3 to cpu 11
igb3: <Intel(R) PRO/1000 Network Connection version - 2.3.10> port 0x2000-0x201f mem 0xc6400000-0xc641ffff,0xc5c00000-0xc5ffffff,0xc64c0000-0xc64c3fff irq 25 at device 0.1 on pci
3
igb3: Using MSIX interrupts with 5 vectors
igb3: Ethernet address: 00:1b:21:4a:69:79
igb3: Bound queue 0 to cpu 12
igb3: Bound queue 1 to cpu 13
igb3: Bound queue 2 to cpu 14
igb3: Bound queue 3 to cpu 15
09000 546827 20995102 deny ip from any to 224.0.0.0/8
09900 251418446 34849277439 fwd 127.0.0.1,3333 tcp from table(100) to not table(9) dst-port 80
09901 251226827 74150859375 allow tcp from any 80 to table(100) out
09999 324676485 22931487657 deny ip from not table(9) to table(100)
09999 93075888 5276322115 deny ip from table(100) to not table(9)
10000 234714177213 241730704799083 allow ip from table(5) to any
10005 245356169 18235355072 deny ip from any to any dst-port 135,137-139,445 out
10006 2929342953 182985124889 deny ip from table(104) to any
10020 688240709 620932403164 divert 8668 ip from any to 1.1.1.1
10400 682416642 620798165276 allow ip from any to any diverted
10770 73183544 9041870946 deny ip from table(2) to any out via vlan18
10772 11698 802274 deny ip from table(3) to any out via vlan4
10773 8807403 463870927 deny ip from any to table(2) out iptos reliability
10774 4923414 300617694 deny ip from any to table(3) out iptos reliability
10775 99485 4397077 deny ip from any to table(3) out iptos throughput
11010 3659429 430047150 deny ip from any to any not verrevpath in via vlan6
11020 719931 58619220 deny ip from any to any not verrevpath in via vlan7
11025 68141 5144481 deny ip from any to any not verrevpath in via vlan8
11030 202144 6785732 deny ip from any to any not verrevpath in via vlan9
11040 171291 56196945 deny ip from any to any not verrevpath in via vlan10
11045 291914032 39427773226 deny ip from any to any not verrevpath in via vlan11
11060 6102962 441745213 deny ip from any to any not verrevpath in via vlan15
11070 4832442 1259880158 deny ip from any to any not verrevpath in via vlan16
11080 814769 95745079 deny ip from any to any not verrevpath in via vlan17
11101 2901098 628552748 deny ip from any to any not verrevpath in via vlan26
11102 1264750 146468688 deny ip from any to any not verrevpath in via vlan27
11110 902441 294155831 deny ip from any to any not verrevpath in via vlan21
11120 628324 31060933 deny ip from any to any not verrevpath in via vlan23
11130 1381 83245 deny ip from any to any not verrevpath in via vlan24
11138 4258607 3389925416 deny ip from any to any not verrevpath in via vlan31
11150 56 2792 deny ip from any to any not verrevpath in via vlan40
15000 3363576 188412499 deny ip from not table(30) to table(31) out
19950 64832991 3461330324 deny tcp from table(25) to not table(8) dst-port 25 out
19960 693595 34424883 deny ip from table(101) to table(103) out
19970 466690 57539243 deny ip from not table(30) to me dst-port 161,162,21,3306
20000 35523656903 32569055261754 pipe tablearg ip from any to table(1) out iptos reliability
20010 36208900912 9635678183009 pipe tablearg ip from table(6) to any out via vlan18
20020 6963415930 5823875049163 pipe tablearg ip from any to table(10) out
20030 5370808609 1175572076679 pipe tablearg ip from table(11) to any out
60005 3749710 1625777707 deny udp from any to 2.2.2.100 dst-port 5060
60005 7940451 2910219814 deny udp from any to 2.2.2.1 dst-port 5060
60020 578206 71125954 divert 8668 ip from 192.168.0.0/16 to any out via vlan4
60020 120740 17363073 divert 8668 ip from 192.168.0.0/16 to any out via vlan5
60020 6485285 2421107818 divert 8668 ip from 192.168.0.0/16 to any out via vlan18
60020 22096 1876197 divert 8668 ip from 192.168.0.0/16 to any out via vlan11
60600 529456103 183816441399 allow ip from any to any diverted
62110 2482047796 207871928397 deny ip from not table(32) to any out via vlan18
62120 34184526 40243097237 allow ip from 3.3.3.0/24 to 3.3.3.0/24 via vlan4
62130 19323045 1282467423 deny ip from not table(32) to any out via vlan4
62140 21168902 1790816969 deny ip from any to not table(32) in via vlan4
64000 8160465887601 5338926261446363 allow ip from any to any
65000 1165747 214509370 allow ip from any to any
65535 5625 3645710 deny ip from any to any
AVC> Can you share your traffic rate (e.g. netstat -i -w1), cpu info and NIC
AVC> info?
Now it's:
# netstat -i -w1
input (Total) output
packets errs idrops bytes packets errs bytes colls
312136 0 0 216478043 312375 0 216359751 0
311760 0 0 217559784 311654 0 217792531 0
295196 0 0 203318550 295319 0 211926680 0
300204 0 0 206880841 300219 0 206348483 0
297019 0 0 203171215 296930 0 207103301 0
308142 0 0 211553806 308294 0 207969407 0
320911 0 0 221584256 320955 0 218811245 0
CPU: Intel(R) Xeon(R) CPU E5520 @ 2.27GHz (2261.30-MHz 686-class CPU)
AVC> What does system load (without verrevpath) looks like in comparison with
AVC> 9.0 (in terms of CPU _and_ packets/sec) ?
Sorry, cannot compare it. Old graphs are lost. AFAIR it was up to 30 LA
in peak times when there was about 400+ kpss in and same out. I can
try to add some rules with verrevpath now in 9.2 system.
Without verrevpath rules top ISHP shows:
last pid: 58440; load averages: 2.52, 2.52, 2.51 up 1+06:25:38 14:05:02
268 processes: 17 running, 177 sleeping, 74 waiting
CPU 0: 0.0% user, 0.0% nice, 0.0% system, 28.2% interrupt, 71.8% idle
CPU 1: 0.0% user, 0.0% nice, 0.0% system, 38.0% interrupt, 62.0% idle
CPU 2: 0.4% user, 0.0% nice, 0.8% system, 29.8% interrupt, 69.0% idle
CPU 3: 0.0% user, 0.0% nice, 0.4% system, 26.7% interrupt, 72.9% idle
CPU 4: 0.0% user, 0.0% nice, 0.8% system, 32.5% interrupt, 66.7% idle
CPU 5: 0.0% user, 0.0% nice, 0.8% system, 31.4% interrupt, 67.8% idle
CPU 6: 0.0% user, 0.0% nice, 0.0% system, 30.2% interrupt, 69.8% idle
CPU 7: 0.0% user, 0.0% nice, 0.0% system, 32.2% interrupt, 67.8% idle
CPU 8: 0.0% user, 0.0% nice, 0.8% system, 0.0% interrupt, 99.2% idle
CPU 9: 0.8% user, 0.0% nice, 0.0% system, 0.0% interrupt, 99.2% idle
CPU 10: 0.4% user, 0.0% nice, 1.2% system, 0.0% interrupt, 98.4% idle
CPU 11: 0.0% user, 0.0% nice, 0.0% system, 0.8% interrupt, 99.2% idle
CPU 12: 0.4% user, 0.0% nice, 0.0% system, 0.8% interrupt, 98.8% idle
CPU 13: 0.0% user, 0.0% nice, 0.4% system, 0.0% interrupt, 99.6% idle
CPU 14: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
CPU 15: 0.0% user, 0.0% nice, 0.8% system, 0.0% interrupt, 99.2% idle
netstat -iw 1
input (Total) output
packets errs idrops bytes packets errs bytes colls
322k 0 0 219M 322k 0 220M 0
324k 0 0 224M 324k 0 222M 0
325k 0 0 227M 325k 0 227M 0
352k 0 0 247M 352k 0 242M 0
After adding verrevpath rules:
last pid: 58471; load averages: 3.19, 2.82, 2.64 up 1+06:30:04 14:09:28
270 processes: 21 running, 179 sleeping, 70 waiting
CPU 0: 0.0% user, 0.0% nice, 0.4% system, 51.4% interrupt, 48.2% idle
CPU 1: 0.0% user, 0.0% nice, 0.4% system, 44.7% interrupt, 54.9% idle
CPU 2: 0.0% user, 0.0% nice, 0.8% system, 37.6% interrupt, 61.6% idle
CPU 3: 0.0% user, 0.0% nice, 0.0% system, 38.8% interrupt, 61.2% idle
CPU 4: 0.4% user, 0.0% nice, 0.0% system, 38.8% interrupt, 60.8% idle
CPU 5: 0.0% user, 0.0% nice, 0.4% system, 41.2% interrupt, 58.4% idle
CPU 6: 0.4% user, 0.0% nice, 0.4% system, 43.9% interrupt, 55.3% idle
CPU 7: 0.0% user, 0.0% nice, 0.0% system, 41.6% interrupt, 58.4% idle
Looks like now this rules does not affect load such a way it was
before. But now NICs configuration differs. There were
ifconfig_lagg0="laggproto loadbalance laggport igb0 laggport igb1 laggport igb2 laggport igb3"
and all vlans over lagg0. Now it is one ix0 without lagg and all vlans
are over ix0.
---
Denis V. Klimkov