Freebsd IP Forwarding performance (question,
and some info) [7-stable, current, em, smp]
Paul
paul at gtcomm.net
Thu Jul 3 06:45:59 UTC 2008
Preliminary 32 bit results...
When I started out it looked like 32 bit was worse than 64 bit, but it's
just the timers are different.
For instance, 4000 hz in 64 bit gives better results than 4000hz in 32 bit.
Low HZ gives better result with polling on in 32 bit
Bottom line, so far I'm not able to get any better performance out of 32
bit at all. In fact I think it might be even a tad slower. I didn't see
as high of bursts like I did on 64 bit so far but I'm still testing.
Tomorrow comes opteron 2222 so it's 1ghz faster than this one, and I can
see if it scales directly with cpu speed or what happens.
I did another SMP test with an interesting results. I took one of the
cpus out of the machine, so it was just left with a single 2212 (dual core)
and it performed better. Less contention I suppose?
some results:
kern.hz=4000
hw.em.rxd=512
hw.em.txd=512
polling on, idle polling on (only way I can get a reliable netstat output)
input (em0) output
packets errs bytes packets errs bytes colls
681961 117612 42281586 1 0 226 0
655095 83418 40615892 2 0 220 0
683881 93559 42400626 1 0 178 0
683637 90452 42385498 1 0 178 0
683345 87471 42367394 1 0 178 0
682737 81483 42329696 2 0 220 0
683154 95413 42355552 1 0 178 0
684556 111013 42442476 1 0 178 0
684365 110960 42430634 1 0 178 0
679089 116440 42103518 3 0 534 0
684328 122713 42428340 1 0 178 0
684852 121387 42460828 1 0 178 0
685358 113256 42492200 1 0 178 0
685060 123110 42473724 1 0 178 0
684463 118335 42436710 1 0 178 0
677182 127788 41985300 2 0 356 0
685920 126144 42527044 1 0 178 0
684946 107034 42466656 1 0 178 0
(reboot)
kern.hz=1000
input (em0) output
packets errs bytes packets errs bytes colls
679611 97394 42136046 5 0 762 0
663939 104714 41164254 5 0 1322 0
685538 91102 42503412 4 0 536 0
676704 94629 41955668 2 0 404 0
685323 115060 42490030 1 0 178 0
675954 105506 41909164 2 0 356 0
655321 92118 40629906 1 0 178 0
686826 85674 42583228 2 0 356 0
686378 89983 42555440 1 0 178 0
685539 80180 42503422 1 0 178 0
686704 88626 42575652 1 0 178 0
686567 88596 42567158 1 0 178 0
687031 82640 42595936 3 0 398 0
sysctl -w kern.polling.each_burst=50
kern.polling.each_burst: 256 -> 50
[root at ircrouter ~]# netstat -w1 -I em0
input (em0) output
packets errs bytes packets errs bytes colls
693036 39992 42968315 3 0 400 0
695538 58189 43123360 1 0 178 0
692670 62765 42945544 1 0 178 0
693219 60755 42979580 2 0 220 0
692637 64761 42943498 1
sysctl -w kern.polling.each_burst=33
kern.polling.each_burst: 50 -> 33
[root at ircrouter ~]# netstat -w1 -I em0
input (em0) output
packets errs bytes packets errs bytes colls
690530 63359 42812868 1 0 226 0
689748 57670 42764380 1 0 178 0
690489 57874 42810322 1 0 178 0
689655 60606 42758614 1 0 178 0
^C
[root at ircrouter ~]# sysctl -w kern.polling.each_burst=3
kern.polling.each_burst: 33 -> 3
[root at ircrouter ~]# netstat -w1 -I em0
input (em0) output
packets errs bytes packets errs bytes colls
612234 110896 37958512 1 0 226 0
614391 112506 38092246 1 0 178 0
^C
[root at ircrouter ~]# sysctl -w kern.polling.each_burst=800
kern.polling.each_burst: 3 -> 800
[root at ircrouter ~]# netstat -w1 -I em0
input (em0) output
packets errs bytes packets errs bytes colls
668057 76496 41419538 1 0 226 0
667689 88674 41396720 2 0 220 0
670526 106654 41572616 1 0 178 0
667326 97832 41374216 1 0 178 0
^C
[root at ircrouter ~]# sysctl -w kern.polling.each_burst=66
kern.polling.each_burst: 800 -> 66
[root at ircrouter ~]# netstat -w1 -I em0
input (em0) output
packets errs bytes packets errs bytes colls
690164 89290 42790172 1 0 226 0
688886 74360 42710936 1 0 178 0
674079 77027 41792902 1 0 178 0
kern.hz=2000
input (em0) output
packets errs bytes packets errs bytes colls
699116 238016 43345196 1 0 178 0
698263 225244 43292310 1 0 290 0
697246 222395 43229256 1 0 178 0
696749 207766 43198442 1 0 178 0
697304 217384 43232852 1 0 178 0
696401 209901 43176866 1 0 178 0
696508 207757 43183500 1 0 178 0
^C
hz=2000 with 1024/1024 descriptors
input (em0) output
packets errs bytes packets errs bytes colls
670315 235780 41559534 1 0 226 0
683218 225838 42359520 1 0 178 0
682998 242551 42345880 1 0 178 0
681777 239649 42270178 1 0 178 0
hz=1000 with 256/256 descriptors
netstat -w1 -I em0
input (em0) output
packets errs bytes packets errs bytes colls
740584 160355 45916212 2 0 0 0
746027 165198 46253678 1 0 178 0
746068 165921 46256220 1 0 178 0
746505 167527 46283314 1 0 178 0
743902 175019 46121928 1 0 178 0
746130 179795 46260064 1 0 178 0
744457 166448 46156338 1 0 178 0
746169 176137 46262482 1 0 178 0
hz=667 with 256/256
input (em0) output
packets errs bytes packets errs bytes colls
742614 91687 46042072 1 0 226 0
739746 85695 45864256 1 0 178 0
733723 85162 45490840 3 0 398 0
737561 102207 45728786 1 0 178 0
739618 127597 45856320 1 0 178 0
^C
Hrm finally same pps as 64 bit.....
Now I wonder what happens if I go back to the 64 bit and try 1000
256/256 ?? I don't think I tried that one..
Guess another reinstall :>
Installing 64 bit.. (again) Just to be sure..
Paul
Ingo Flaschberger wrote:
> Dear Paul,
>
>> SMP DISABLED on my Opteron 2212 (ULE, Preemption on)
>> Yields ~750kpps in em0 and out em1 (one direction)
>> I am miffed why this yields more pps than
>> a) with all 4 cpus running and b) 4 cpus with lagg load balanced over
>> 3 incoming connections so 3 taskq threads
>
> because less locking, less synchronisation, ....
>
>> I would be willing to set up test equipment (several servers plugged
>> into a switch) with ipkvm and power port access
>> if someone or a group of people want to figure out ways to improve
>> the routing process, ipfw, and lagg.
>>
>> Maximum PPS with one ipfw rule on UP:
>> tops out about 570Kpps.. almost 200kpps lower ? (frown)
>
> can you post the rule here?
>
>> I'm going to drop in a 3ghz opteron instead of the 2ghz 2212 that's
>> in here and see how that scales, using UP same kernel etc I have now.
>
> really, please try 32bit and 1 cpu.
>
> Kind regards,
> Ingo Flaschberger
>
> _______________________________________________
> freebsd-net at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>
More information about the freebsd-net
mailing list