Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

[Paul]

With hours and days of tweaking i can't even get 500k pps :/ no firewall 
no anything else..
What is your kernel config? Sysctl configs?
My machine i'm testing on is dual opteron 2212  , with intel 2 port 
82571 nic..  Using 7-STABLE and I tried 6-stable and -current
I get the RTM_MISS with 7 and current but only with certain types of 
packets at a certain rate.. :/
I can not get more than 500kpps.. i tried everything I could think of... 
lowering the rx descriptors on EM to 512 instead of 2048 gave me some 
more.. I was stuck at 400kpps until i changed those and i lowered the rx 
processing limit.   
My tests are going incoming em0 and outgoing em1  in one direction only 
and it has major errors when em0 taskq gets close to 80% cpu..
I am pretty disappointed that it maxes out a little over 400kpps and 
even then it gets some errors here and there , mainly missed packets due 
to no buffer and rx overruns (dev.em.0.stats=1)



Mike Tancsa wrote:
> At 04:04 AM 6/29/2008, Paul wrote:
>> This is just a question but who can get more than 400k pps forwarding 
>> performance ?
>
>
> OK, I setup 2 boxes on either end of a RELENG_7 box from about May 7th 
> just now, to see with 2 boxes blasting across it how it would work.  
> *However*, this is with no firewall loaded and, I must enable ip fast 
> forwarding. Without that enabled, the box just falls over.
>
> even at 20Kpps, I start seeing all sorts of messages spewing to route 
> -n monitor
>
>
> got message of size 96 on Mon Jun 30 15:39:10 2008
> RTM_MISS: Lookup failed on this address: len 96, pid: 0, seq 0, errno 
> 0, flags:<DONE>
> locks:  inits:
> sockaddrs: <DST>
>  default
>
> I am starting to wonder if those messages are the results of corrupted 
> packets the machine just cant keep up with ?
>
>
> CPU is
>
> CPU: Intel(R) Xeon(R) CPU            3070  @ 2.66GHz (2660.01-MHz 
> 686-class CPU)
>
>
>            input        (Total)           output
>    packets  errs      bytes    packets  errs      bytes colls
>     611945     0   77892098     611955     0   77013002     0
>     616727     0   78215508     616742     0   77303454     0
>     617066     0   78162130     617082     0   77238434     0
>     618238     0   78302314     618225     0   77377582     0
>     617035     0   78141000     617038     0   77215672     0
>     617625     0   78225600     617588     0   77301734     0
>     616190     0   78017320     616165     0   77091774     0
>     615583     0   78064130     615628     0   77152800     0
>     617662     0   78254388     617658     0   77332340     0
>     618000     0   78269912     617950     0   77344554     0
>     617248     0   78183136     617315     0   77259588     0
>     617325     0   78204566     617289     0   77282094     0
>     618391     0   78337734     618357     0   77413756     0
>     616025     0   78116070     616082     0   77203116     0
>
>
> To generate the packets, I am just using 
> /usr/src/tools/tools/netblast  on 2 endpoints starting at about the 
> same time
>
> # ./netblast 10.10.1.2 500 100 40
>
> start:             1214854131.083679919
> finish:            1214854171.084668592
> send calls:        20139141
> send errors:       0
> approx send rate:  503478
> approx error rate: 0
>
>
> # ./netblast 10.10.1.3 500 10 40
>
> start:             1214854273.882202815
> finish:            1214854313.882319031
> send calls:        23354971
> send errors:       18757223
> approx send rate:  114943
> approx error rate: 0
>
> The box in the middle doing the forwarding
>
> 1[spare-r7]# ifconfig -u
> em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
>         
> options=19b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4>
>         ether 00:1b:21:08:32:a8
>         inet 10.20.1.1 netmask 0xffffff00 broadcast 10.20.1.255
>         media: Ethernet autoselect (1000baseTX <full-duplex>)
>         status: active
> em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
>         options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
>         ether 00:1b:21:08:32:a9
>         inet 192.168.43.193 netmask 0xffffff00 broadcast 192.168.43.255
>         media: Ethernet autoselect (100baseTX <full-duplex>)
>         status: active
> em3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
>         
> options=19b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4>
>         ether 00:30:48:90:4c:ff
>         inet 10.10.1.1 netmask 0xffffff00 broadcast 10.10.1.255
>         media: Ethernet autoselect (1000baseTX <full-duplex>)
>         status: active
> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
>         inet 127.0.0.1 netmask 0xff000000
>
>
> I am going to try a few more tests with and without, firewall rules 
> etc as well as an updated kernel to RELENG_7 as of today and see how 
> that goes.
>
>         ---Mike
>
>