Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

[Andre Oppermann]

Bruce Evans wrote:
> On Mon, 7 Jul 2008, Andre Oppermann wrote:
> 
>> Ingo Flaschberger wrote:
>>> I don't think you will be able to route 64byte packets at 1gbit 
>>> wirespeed (2Mpps) with a current x86 platform.
>>
>> You have to take inter-frame gap and other overheads too.  That gives
>> about 1.244Mpps max on a 1GigE interface.
> 
> What are the other overheads?  I calculate 1.644Mpps counting the 
> inter-frame
> gap, with 64-byte packets and 64-header_size payloads.  If the 64 bytes
> is for the payload, then the max is much lower.

The theoretical maximum at 64byte frames is 1,488,100.  I've looked
up my notes the 1.244Mpps number can be ajusted to 1.488Mpps.

>>> I hoped to reach 1Mpps with the hardware I mentioned some mails 
>>> before, but 2Mpps is far far away.
>>> Currently I get 160kpps via pci-32mbit-33mhz-1,2ghz mobile pentium.
>>
>> This is more or less expected.  PCI32 is not able to sustain high
>> packet rates.  The bus setup times kill the speed.  For larger packets
>> the ratio gets much better and some reasonable throughput can be 
>> achieved.
> 
> I get about 640 kpps without forwarding (sendto: slightly faster;
> recvfrom: slightly slower) on a 2.2GHz A64.  Underclocking the memory
> from 200MHz to 100MHz only reduces the speed by about 10%, while not
> overclocking the CPU by 10% reduces the speed by the same 10%, so the
> system is apparently still mainly CPU-bound.

On PCI32 at 33MHz?  He's using a 1.2GHz Mobile Pentium on top of that.

>> NetFPGA doesn't have enough TCAM space to be useful for real routing
>> (as in Internet sized routing table).  The trick many embedded networking
>> CPUs use is cache prefetching that is integrated with the network
>> controller.  The first 64-128bytes of every packet are transferred
>> automatically into the L2 cache by the hardware.  This allows relatively
>> slow CPUs (700 MHz Broadcom BCM1250 in Cisco NPE-G1 or 1.67-GHz Freescale
>> 7448 in NPE-G2) to get more than 1Mpps.  Until something like this is
>> possible on Intel or AMD x86 CPUs we have a ceiling limited by RAM speed.
> 
> Does using fa$ter memory (speed and/or latency) help here?  64 bytes
> is so small that latency may be more of a problem, especially without
> a prefetch.

Latency.  For IPv4 packet forwarding only one cache line per packet
is fetched.  More memory speed only helps with the DMA from/to the
network card.

-- 
Andre