Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

[Paul]

SMP DISABLED on my Opteron 2212  (ULE, Preemption on)
Yields ~750kpps in em0 and out em1  (one direction)
I am miffed why this yields more pps than
a) with all 4 cpus running and b) 4 cpus with lagg load balanced over 3 
incoming connections so 3 taskq threads

I would be willing to set up test equipment (several servers plugged 
into a switch) with ipkvm and power port access
if someone or a group of people want to figure out ways to improve the 
routing process, ipfw, and lagg.

Maximum PPS with one ipfw rule on UP:
tops out about 570Kpps.. almost 200kpps lower ? (frown)

I'm going to drop in a 3ghz opteron instead of the 2ghz 2212 that's in 
here and see how that scales, using UP same kernel etc I have now.





Julian Elischer wrote:
> Paul wrote:
>> ULE without PREEMPTION is now yeilding better results.
>>         input          (em0)           output
>>   packets  errs      bytes    packets  errs      bytes colls
>>    571595 40639   34564108          1     0        226     0
>>    577892 48865   34941908          1     0        178     0
>>    545240 84744   32966404          1     0        178     0
>>    587661 44691   35534512          1     0        178     0
>>    587839 38073   35544904          1     0        178     0
>>    587787 43556   35540360          1     0        178     0
>>    540786 39492   32712746          1     0        178     0
>>    572071 55797   34595650          1     0        178     0
>>  
>> *OUCH, IPFW HURTS..
>> loading ipfw, and adding one ipfw rule allow ip from any to any drops 
>> 100Kpps off :/ what's up with THAT?
>> unloaded ipfw module and back 100kpps more again, that's not right 
>> with ONE rule.. :/
>
> ipfw need sto gain a lock on hte firewall before running,
> and is quite complex..  I can believe it..
>
> in FreeBSD 4.8 I was able to use ipfw and filter 1Gb between two 
> interfaces (bridged) but I think it has slowed down since then due to 
> the SMP locking.
>
>
>>
>> em0 taskq is still jumping cpus.. is there any way to lock it to one 
>> cpu or is this just a function of ULE
>>
>> running a tar czpvf all.tgz *  and seeing if pps changes..
>> negligible.. guess scheduler is doing it's job at least..
>>
>> Hmm. even when it's getting 50-60k errors per second on the interface 
>> I can still SCP a file through that interface although it's not 
>> fast.. 3-4MB/s..
>>
>> You know, I wouldn't care if it added 5ms latency to the packets when 
>> it was doing 1mpps as long as it didn't drop any.. Why can't it do 
>> that? Queue them up and do them in bigggg chunks so none are 
>> dropped........hmm?
>>
>> 32 bit system is compiling now..  won't do > 400kpps with GENERIC 
>> kernel, as with 64 bit did 450k with GENERIC, although that could be
>> the difference between opteron 270 and opteron 2212..
>>
>> Paul
>>
>> _______________________________________________
>> freebsd-net at freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>
>