net.inet.ip.forwarding and net.inet.ip.fastforwarding

[Robert Watson]

On Fri, 9 Sep 2005, Andre Oppermann wrote:

>> 6.0 and 7.x share the same code so the settings are identical.  As to 
>> downside you pay a penalty if the fastforwarding code has to hand the 
>> packet back to the "slow path".  There may also be side effects from 
>> the run-to-completion model it uses.  You should test to decide if the 
>> feature is worth enabling for your environment.  I'm not sure it's had 
>> much testing (Andre?).
>
> When activated on a router it gives a very nice speed boost.  Process 
> completion pays off very well here.  It has got a lot of testing at 
> various ISP's on their production routers.  For hosts it doesn't really 
> hurt but is totally pointless.

In measurements a couple of years ago, I demonstrated to myself that on 
several interesting pieces of hardware, running with net.isr.enable=1 
resulted in lower latency packet forwarding and processing on 5.x (at the 
time) than 4.x (at the time).  I've not re-measured with recent 7.x/6.x or 
4.x on recent hardware.  Over the last couple of years, we've shaken out a 
number of important bugs in local network stack code that tripped up with 
net.isr.enable, so we're reaching the point where I might start 
encouraging people to work with it more actively for local (as well as 
routed) paths.

There are still open questions about what models make the most sense, 
though -- run to completion has some nice latency properties, and also 
increases the opportunities for parallelism in the network stack.  On the 
other hand, it increases the load born by ithreads, so if your ithread was 
already maxing out available CPU, you would decrease the amount of work it 
could do, and on UP it can result in more context switches if you have 
several active interfaces running out of different ithreads.  Many of 
these questions are the same as the ones we'll be talking about for next 
generation polling support at the developer summit this fall, so it would 
make sense to talk about them at the same time.

Robert N M Watson