Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp]

[Bruce Evans]

On Mon, 7 Jul 2008, Robert Watson wrote:

> Since you're doing fine-grained performance measurements of a code path that 
> interests me a lot, could you compare the cost per-send on UDP for the 
> following four cases:
>
> (1) sendto() to a specific address and port on a socket that has been bound 
> to
>    INADDR_ANY and a specific port.
>
> (2) sendto() on a specific address and port on a socket that has been bound 
> to
>    a specific IP address (not INADDR_ANY) and a specific port.
>
> (3) send() on a socket that has been connect()'d to a specific IP address and
>    a specific port, and bound to INADDR_ANY and a specific port.
>
> (4) send() on a socket that has been connect()'d to a specific IP address
>    and a specific port, and bound to a specific IP address (not INADDR_ANY)
>    and a specific port.
>
> The last of these should really be quite a bit faster than the first of 
> these, but I'd be interested in seeing specific measurements for each if 
> that's possible!

Not sure if I understand networking well enough to set these up quickly.
Does netrate use one of (3) or (4) now?

I can tell you vaguely about old results for netrate (send()) vs ttcp
(sendto()).  send() is lighter weight of course, and this made a difference
of 10-20%, but after further tuning the difference became smaller, which
suggests that everything ends up waiting for something in common.

Now I can measure cache misses better and hope that a simple count of
cache misses will be a more reproducible indicator of significant
bottlenecks than pps.  I got nowhere trying to reduce instruction
counts, possibly because it would take avoiding 100's of instructions
to get the same benefit as avoiding a single cache miss.

Bruce