Some performance measurements on the FreeBSD network stack

[Luigi Rizzo]

On Thu, Apr 19, 2012 at 11:06:38PM +0200, K. Macy wrote:
> On Thu, Apr 19, 2012 at 11:22 PM, Luigi Rizzo <rizzo at iet.unipi.it> wrote:
> > On Thu, Apr 19, 2012 at 10:34:45PM +0200, K. Macy wrote:
> >> >> This is indeed a big problem. ?I'm working (rough edges remain) on
> >> >> changing the routing table locking to an rmlock (read-mostly) which
> >> >
> >>
> >> This only helps if your flows aren't hitting the same rtentry.
> >> Otherwise you still convoy on the lock for the rtentry itself to
> >> increment and decrement the rtentry's reference count.
> >>
> >> > i was wondering, is there a way (and/or any advantage) to use the
> >> > fastforward code to look up the route for locally sourced packets ?
> >
> > actually, now that i look at the code, both ip_output() and
> > the ip_fastforward code use the same in_rtalloc_ign(...)
> >
> >> >
> >>
> >> If the number of peers is bounded then you can use the flowtable. Max
> >> PPS is much higher bypassing routing lookup. However, it doesn't scale
> >> to arbitrary flow numbers.
> >
> > re. flowtable, could you point me to what i should do instead of
> > calling in_rtalloc_ign() ?
> 
> If you build with it in your kernel config and enable the sysctl
> ip_output will automatically use it for TCP and UDP connections. If
> you're doing forwarding you'll need to patch the forwarding path.

cool.
For the records, with "netsend 10.0.0.2 ports 18 0 5" on an ixgbe
talking to a remote host i get the following results (with a single
port netsend does a connect() and then send(), otherwise it
loops around a sendto() )

	net.flowtable.enabled	port          	ns/pkt
	-----------------------------------------------------
	not compiled in		5000		 944	M_FLOWID not set
	0 (disable)		5000		1004
	1 (enable)		5000		 980

	not compiled in		5000-5001	3400	M_FLOWID not set
	0 (disable)		5000-5001	1418
	1 (enable)		5000-5001	1230

The small penalty when flowtable is disabled but compiled in is
probably because the net.flowtable.enable flag is checked
a bit deep in the code.

The advantage with non-connect()ed sockets is huge. I don't
quite understand why disabling the flowtable still helps there.

cheers
luigi