Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

[Paul]

Apparently lagg hasn't been giant fixed :/   Can we do something about 
this quickly? 
with adaptive giant i get more performance on lagg but the cpu usage is 
smashed 100%
I get about 50k more pps per interface  (so 150kpps total which STILL is 
less than a single gigabit port)
Check it out

68 processes:  9 running, 41 sleeping, 18 waiting
CPU:  0.0% user,  0.0% nice, 89.5% system,  0.0% interrupt, 10.5% idle
Mem: 8016K Active, 6192K Inact, 47M Wired, 108K Cache, 9056K Buf, 1919M Free
Swap: 8192M Total, 8192M Free

  PID USERNAME PRI NICE   SIZE    RES STATE  C   TIME   WCPU COMMAND
   38 root     -68    -     0K    16K CPU1   1   3:29 100.00% em2 taskq
   37 root     -68    -     0K    16K CPU0   0   3:31 98.78% em1 taskq
   36 root     -68    -     0K    16K CPU3   3   2:53 82.42% em0 taskq
   11 root     171 ki31     0K    16K RUN    2  22:48 79.00% idle: cpu2
   10 root     171 ki31     0K    16K RUN    3  20:51 22.90% idle: cpu3
   39 root     -68    -     0K    16K RUN    2   0:32 16.60% em3 taskq
   12 root     171 ki31     0K    16K RUN    1  20:16  2.05% idle: cpu1
   13 root     171 ki31     0K    16K RUN    0  20:25  1.90% idle: cpu0

            input          (em0)           output
   packets  errs      bytes    packets  errs      bytes colls
    122588     0    7355280          0     0          0     0
    123057     0    7383420          0     0          0     0

            input          (em1)           output
   packets  errs      bytes    packets  errs      bytes colls
    174917 11899   10495032          2     0        178     0
    173967 11697   10438038          2     0        356     0
    174630 10603   10477806          2     0        268     0

            input          (em2)           output
   packets  errs      bytes    packets  errs      bytes colls
    175843  3928   10550580          0     0          0     0
    175952  5750   10557120          0     0          0     0


Still less performance than single gig-e.. that giant lock really sucks 
, and why on earth would LAGG require that.. It  seems so simple to fix :/
Anyone up for it:) I wish I was a programmer sometimes, but network 
engineering will have to do. :D



Julian Elischer wrote:
> Paul wrote:
>> Is PF better than ipfw?  iptables almost has no impact on routing 
>> performance unless I add a swath of rules to it and then it bombs
>> I need maybe 10 rules max  and I don't want 20% performance drop for 
>> that.. :P
>
> well lots of people have wanted to fix it, and I've investigated
> quite a lot but it takes someone with 2 weeks of free time and
> all the right clue. It's not inherrent in ipfw but it needs some
> TLC from someone who cares :-).
>
>
>
>> Ouch! :)  Is this going to be fixed any time soon?  We have some 
>> money that can be used for development costs to fix things like this 
>> because
>> we use linux and freebsd machines as firewalls for a lot of customers 
>> and with the increasing bandwidth and pps the customers are demanding 
>> more and I
>> can't give them better performance with a brand new dual xeon or 
>> opteron machine vs the old p4 machines I have them running on now :/  
>> The only difference
>> in the new machine vs old machine is that the new one can take in 
>> more pps and drop it but it can't route a whole lot more.  
>> Routing/firewalling must still not be lock free, ugh.. :P
>>
>> Thanks
>>
>>
>>
>> Julian Elischer wrote:
>>> Paul wrote:
>>>> ULE without PREEMPTION is now yeilding better results.
>>>>         input          (em0)           output
>>>>   packets  errs      bytes    packets  errs      bytes colls
>>>>    571595 40639   34564108          1     0        226     0
>>>>    577892 48865   34941908          1     0        178     0
>>>>    545240 84744   32966404          1     0        178     0
>>>>    587661 44691   35534512          1     0        178     0
>>>>    587839 38073   35544904          1     0        178     0
>>>>    587787 43556   35540360          1     0        178     0
>>>>    540786 39492   32712746          1     0        178     0
>>>>    572071 55797   34595650          1     0        178     0
>>>>  
>>>> *OUCH, IPFW HURTS..
>>>> loading ipfw, and adding one ipfw rule allow ip from any to any 
>>>> drops 100Kpps off :/ what's up with THAT?
>>>> unloaded ipfw module and back 100kpps more again, that's not right 
>>>> with ONE rule.. :/
>>>
>>> ipfw need sto gain a lock on hte firewall before running,
>>> and is quite complex..  I can believe it..
>>>
>>> in FreeBSD 4.8 I was able to use ipfw and filter 1Gb between two 
>>> interfaces (bridged) but I think it has slowed down since then due 
>>> to the SMP locking.
>>>
>>>
>>>>
>>>> em0 taskq is still jumping cpus.. is there any way to lock it to 
>>>> one cpu or is this just a function of ULE
>>>>
>>>> running a tar czpvf all.tgz *  and seeing if pps changes..
>>>> negligible.. guess scheduler is doing it's job at least..
>>>>
>>>> Hmm. even when it's getting 50-60k errors per second on the 
>>>> interface I can still SCP a file through that interface although 
>>>> it's not fast.. 3-4MB/s..
>>>>
>>>> You know, I wouldn't care if it added 5ms latency to the packets 
>>>> when it was doing 1mpps as long as it didn't drop any.. Why can't 
>>>> it do that? Queue them up and do them in bigggg chunks so none are 
>>>> dropped........hmm?
>>>>
>>>> 32 bit system is compiling now..  won't do > 400kpps with GENERIC 
>>>> kernel, as with 64 bit did 450k with GENERIC, although that could be
>>>> the difference between opteron 270 and opteron 2212..
>>>>
>>>> Paul
>>>>
>>>> _______________________________________________
>>>> freebsd-net at freebsd.org mailing list
>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>>>> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>>>
>>>
>
>