Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

[Paul]

> I use low-end memory, but on the machine that does 640 kpps it somehow
> has latency almost 4 times as low as on new FreeBSD cluster machines
> (~42 nsec instead of ~150).  perfmon (fixed for AXP and A64) and hwpmc
> report an average of 11 k8-dc-misses per sendto() while sending via
> bge at 640 kpps.  11 * 42 accounts for 442 nsec out of the 1562 per
> packet at this rate.  11 * 150 = 1650 would probably make this rate
> unachievable despite the system having 20 times as much CPU and bus.
>
Any of the buffered dimms or ddr3 or high cas ddr2 are going to have a 
lot more latency
than older ones because the frequency is so high or the buffering.  The 
best is to use ddr2 with
the lowest timings that it supports at the highest frequency but not the 
highest frequency it supports
at higher timings.. for instance i have some 1100mhz ddr2 ram but it's 
5-5-5-15 but it will do
5-4-4-12 at 1000 or 900 Mhz so I think the latency may have more impact 
on the speed than
the actual MHz of the ram itself.   This works for several benchmarks 
which I have tested before
running the ram at 1:1 with the FSB  (400 FSB(1600fsb actual) with ram 
at 800 and the latency is
a lot lower than ram at 1:1.20 FSB even though the bandwidth is higher)
With higher latency in the 'server' machines we probably need to do 
things in bigger chunks.. Anyone
using a FBSD router isn't going to care about a 1ms delay in the packet 
but they will care if packets are
dropped or reordered.


Paul