netmap in GENERIC, by default, on HEAD
Andrey V. Elsukov
bu7cher at yandex.ru
Wed Nov 5 17:00:41 UTC 2014
On 05.11.2014 19:18, Evandro Nunes wrote:
> On Wed, Nov 5, 2014 at 1:52 PM, Andrey V. Elsukov <bu7cher at yandex.ru> wrote:
>
>> On 05.11.2014 18:39, George Neville-Neil wrote:
>>> Howdy,
>>>
>>> Last night (Pacific Time) I committed a change so that GENERIC, on HEAD
>>> has the netmap
>>> device enabled. This is to increase the breadth of our testing of that
>>> feature prior
>>> to the release of FreeBSD 11.
>>>
>>> In two weeks I will enable IPSec by default, again in preparation for 11.
>>
>> Hi,
>>
>> recently we did some IP forwarding tests and the GENERIC kernel is
>> several times faster than GENERIC+IPSEC. Even when IPSEC has no SA.
>>
>> I didn't do test on vanilla kernel, but our kernel is able forward
>> IPv4/IPv6 on rate close to 8.6 Mpps. The same kernel compiled with IPSEC
>> can forward only 180 kpps. I think this problem should be solved before
>> enabling it in GENERIC.
>>
>
> this forward rate you mention is related to netmap? or usual
> forwarding/fastforwarding? this is a huge number, do you mind sharing your
> dmesg output and top -PSH output so I can check for interrupt CPU usage and
> other relevant stuff?
This is patched kernel without netmap and fastforwarding. We removed all
lock contention on the forwarding path to be sure that it doesn't affect
IPSEC.
Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz (2200.05-MHz K8-class CPU)
FreeBSD/SMP: Multiprocessor System Detected: 32 CPUs
FreeBSD/SMP: 2 package(s) x 8 core(s) x 2 SMT threads
real memory = 68736253952 (65552 MB)
avail memory = 66370662400 (63295 MB)
ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.5.15>
port 0x7020-0x703f mem 0xde680000-0xde6fffff,0xde704000-0xde707fff irq
32 at device 0.0 on pci4
ix0: Using MSIX interrupts with 16 vectors
ix0: Ethernet address: 90:e2:ba:0d:73:54
ix0: PCI Express Bus: Speed 5.0GT/s Width x8
This is IPv6 forwarding test - 6 /64 prefixes each has 200 random
addresses. They are routed between 6 vlans.
# netstat -I ix0 -w 1
input (ix0) output
packets errs idrops bytes packets errs bytes colls
8917043 0 0 571436864 8149880 0 522587200 0
8943391 0 0 571598336 8179318 0 525085504 0
8928155 0 0 571262144 8168254 0 522712192 0
8921342 0 937 571693504 8128132 0 521997184 0
8924322 0 0 571170048 8211500 0 520264320 0
8934564 0 0 571483584 8180040 0 524475264 0
8937039 0 0 571384640 8234779 0 525686080 0
8926528 0 0 571481728 8160380 0 524265920 0
8923160 0 0 571397248 8229839 0 522569408 0
8930070 0 1705 571594944 8216092 0 528481152 0
8916249 0 0 571294784 8184286 0 524399360 0
8937301 0 0 571391040 8221895 0 526383744 0
8927967 0 0 571613312 8164779 0 524997760 0
8936306 0 0 571251712 8167960 0 519575744 0
8922983 0 306 571430528 8216466 0 525893056 0
8916209 0 0 571434240 8202692 0 526046336 0
8945608 0 0 571426624 8265756 0 524815552 0
8925548 0 1045 571444480 8229681 0 530935232 0
8932145 0 0 571747200 8149710 0 523409536 0
8929339 0 0 571683200 8186790 0 520719040 0
8917697 0 0 571585152 8212635 0 525775680 0
# top -PSH
last pid: 2788; load averages: 12.01, 4.76, 1.92
up 0+00:04:38 20:58:48
471 processes: 45 running, 344 sleeping, 82 waiting
CPU 0: 0.0% user, 0.0% nice, 21.6% system, 68.2% interrupt, 10.2% idle
CPU 1: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
CPU 2: 0.0% user, 0.0% nice, 2.7% system, 84.3% interrupt, 12.9% idle
CPU 3: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
CPU 4: 0.0% user, 0.0% nice, 3.9% system, 86.7% interrupt, 9.4% idle
CPU 5: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
CPU 6: 0.0% user, 0.0% nice, 5.5% system, 88.6% interrupt, 5.9% idle
CPU 7: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
CPU 8: 0.0% user, 0.0% nice, 3.5% system, 90.2% interrupt, 6.3% idle
CPU 9: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
CPU 10: 0.0% user, 0.0% nice, 3.1% system, 87.1% interrupt, 9.8% idle
CPU 11: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
CPU 12: 0.0% user, 0.0% nice, 27.5% system, 62.0% interrupt, 10.6% idle
CPU 13: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
CPU 14: 0.0% user, 0.0% nice, 6.3% system, 85.9% interrupt, 7.8% idle
CPU 15: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
CPU 16: 0.0% user, 0.0% nice, 17.6% system, 79.6% interrupt, 2.7% idle
CPU 17: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
CPU 18: 0.0% user, 0.0% nice, 2.4% system, 92.2% interrupt, 5.5% idle
CPU 19: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
CPU 20: 0.0% user, 0.0% nice, 7.8% system, 86.7% interrupt, 5.5% idle
CPU 21: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
CPU 22: 0.0% user, 0.0% nice, 6.3% system, 87.5% interrupt, 6.3% idle
CPU 23: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
CPU 24: 0.0% user, 0.0% nice, 1.6% system, 89.4% interrupt, 9.0% idle
CPU 25: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
CPU 26: 0.0% user, 0.0% nice, 2.0% system, 91.8% interrupt, 6.3% idle
CPU 27: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
CPU 28: 0.0% user, 0.0% nice, 2.7% system, 87.8% interrupt, 9.4% idle
CPU 29: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
CPU 30: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
CPU 31: 0.0% user, 0.0% nice, 0.4% system, 0.0% interrupt, 99.6% idle
Mem: 17M Active, 13M Inact, 716M Wired, 61G Free
ARC: 28M Total, 6102K MFU, 20M MRU, 16K Anon, 373K Header, 1492K Other
Swap: 16G Total, 16G Free
# top -HSIzts1
last pid: 2805; load averages: 13.88, 7.92, 3.56
up 0+00:06:40 21:00:50
469 processes: 45 running, 342 sleeping, 82 waiting
CPU: 0.0% user, 0.0% nice, 3.5% system, 39.3% interrupt, 57.1% idle
Mem: 17M Active, 13M Inact, 716M Wired, 61G Free
ARC: 28M Total, 6105K MFU, 20M MRU, 16K Anon, 373K Header, 1494K Other
Swap: 16G Total, 16G Free
PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
12 root -92 - 0K 1504K CPU22 22 3:38 87.35%
intr{irq293: ix0:que }
12 root -92 - 0K 1504K WAIT 26 3:41 86.57%
intr{irq295: ix0:que }
12 root -92 - 0K 1504K CPU18 18 3:41 86.47%
intr{irq291: ix0:que }
12 root -92 - 0K 1504K CPU28 28 3:41 86.47%
intr{irq296: ix0:que }
12 root -92 - 0K 1504K CPU24 24 3:39 86.18%
intr{irq294: ix0:que }
12 root -92 - 0K 1504K WAIT 4 3:35 86.18%
intr{irq284: ix0:que }
12 root -92 - 0K 1504K CPU8 8 3:35 86.18%
intr{irq286: ix0:que }
12 root -92 - 0K 1504K CPU6 6 3:33 85.60%
intr{irq285: ix0:que }
12 root -92 - 0K 1504K CPU20 20 3:32 85.50%
intr{irq292: ix0:que }
12 root -92 - 0K 1504K CPU10 10 3:32 84.86%
intr{irq287: ix0:que }
12 root -92 - 0K 1504K CPU2 2 3:32 84.67%
intr{irq283: ix0:que }
12 root -92 - 0K 1504K WAIT 14 3:31 84.38%
intr{irq289: ix0:que }
12 root -92 - 0K 1504K CPU16 16 3:11 77.59%
intr{irq290: ix0:que }
12 root -92 - 0K 1504K CPU0 0 2:42 64.70%
intr{irq282: ix0:que }
12 root -92 - 0K 1504K CPU12 12 2:44 63.67%
intr{irq288: ix0:que }
0 root -92 0 0K 4672K - 12 1:01 26.66%
kernel{ix0 que}
0 root -92 0 0K 4672K - 0 0:58 23.00%
kernel{ix0 que}
0 root -92 0 0K 4672K - 16 0:48 18.16%
kernel{ix0 que}
0 root -92 0 0K 4672K - 20 0:20 7.86%
kernel{ix0 que}
0 root -92 0 0K 4672K - 14 0:15 5.57%
kernel{ix0 que}
0 root -92 0 0K 4672K CPU6 6 0:14 5.57%
kernel{ix0 que}
0 root -92 0 0K 4672K - 22 0:12 4.59%
kernel{ix0 que}
0 root -92 0 0K 4672K - 18 0:06 4.59%
kernel{ix0 que}
0 root -92 0 0K 4672K - 28 0:06 4.49%
kernel{ix0 que}
0 root -92 0 0K 4672K - 26 0:06 4.39%
kernel{ix0 que}
0 root -92 0 0K 4672K - 24 0:05 4.05%
kernel{ix0 que}
0 root -92 0 0K 4672K - 8 0:09 3.47%
kernel{ix0 que}
0 root -92 0 0K 4672K - 4 0:08 3.17%
kernel{ix0 que}
0 root -92 0 0K 4672K - 10 0:07 2.98%
kernel{ix0 que}
0 root -92 0 0K 4672K - 2 0:05 2.39%
kernel{ix0 que}
# pmcstat -TS llc-misses -w10
PMC: [llc-misses] Samples: 3949 (100.0%) , 0 unresolved
%SAMP IMAGE FUNCTION CALLERS
28.6 kernel ixgbe_rxeof ixgbe_msix_que:25.9
ixgbe_handle_que:2.8
14.7 kernel bcmp netisr_dispatch_src:13.5
rtalloc_fib_nolock:1.0
5.9 kernel mb_ctor_mbuf uma_zalloc_arg
5.8 kernel ixgbe_mq_start vlan_transmit
4.6 kernel _mtx_trylock ixgbe_mq_start
3.8 kernel ether_nh_input netisr_dispatch_src
2.6 kernel ether_input ixgbe_msix_que
2.4 kernel m_tag_delete_chain m_tag_copy_chain:1.2 uma_zfree_arg:0.9
2.4 kernel netisr_dispatch_src ixgbe_rxeof:1.6 ixgbe_msix_que:0.6
2.2 kernel cpu_search_highest cpu_search_highest
2.2 kernel ixgbe_txeof ixgbe_msix_que
2.1 kernel uma_zfree_arg m_freem
2.0 kernel netisr_dispatch ixgbe_msix_que:1.1 ixgbe_rxeof:0.7
2.0 kernel critical_exit uma_zfree_arg
1.9 kernel _thread_lock_flags ithread_loop:1.2
intr_event_schedule_thread:0.6
1.9 kernel _bus_dmamap_sync ixgbe_rxeof
1.3 kernel ixgbe_xmit ixgbe_mq_start_locked
0.9 kernel ixgbe_msix_que intr_event_execute_handlers
--
WBR, Andrey V. Elsukov
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 538 bytes
Desc: OpenPGP digital signature
URL: <http://lists.freebsd.org/pipermail/freebsd-net/attachments/20141105/cabe00f4/attachment.sig>
More information about the freebsd-net
mailing list