Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

[Robert Watson]

On Mon, 7 Jul 2008, Artem Belevich wrote:

> As was already mentioned, we can't avoid all cache misses as there's data 
> that's recently been updated in memory via DMA and therefor kicked out of 
> cache.
>
> However, we may hide some of the latency penalty by prefetching 
> 'interesting' data early. I.e. we know that we want to access some ethernet 
> headers, so we may start pulling relevant data into cache early. Ideally, by 
> the time we need to access the field, it will already be in the cache. When 
> we're counting nanoseconds per packet this may bring some performance gain.

There were some patches floating around for if_em to do a prefetch of the 
first bit of packet data on packets before handing them up the stack.  My 
understanding is that they moved the hot spot earlier, but didn't make a huge 
difference because it doesn't really take that long to get to the point where 
you're processing the IP header in our current stack (a downside to 
optimization...).  However, that's a pretty anecdotal story, and a proper 
study of the effects of prefetching would be most welcome.  One thing that I'd 
really like to see someone look at is whether, by doing a bit of appropriately 
timed prefetching, we can move cache misses out from under hot locks that 
don't really relate to the data being prefetched.

Robert N M Watson
Computer Laboratory
University of Cambridge