fastforward/routing: a 3 million packet-per-second system?

[Adrian Chadd]

On 28 July 2014 07:51, John Jasen <jjasen at gmail.com> wrote:
> in_input crept up into the top 5, versus fastforward.
>
>
> Would PMC counters help?

Not at the moment. This is a lock contention thing, not a pmc thing. I
bet if you ran pmc the mutex/rwlock things would be up high. :)

>
>  cat debug.lock.pref.stats.out-20140728-1 | sort -nk 4 | tail -5
>        5         4         413         115         160      2      0  0
> 63 /usr/src/sys/kern/kern_condvar.c:145 (sleep mutex:Giant)
>        1         1      148858        4095      650072      0      0  0
> 11184 /usr/src/sys/kern/subr_turnstile.c:552 (spin mutex:turnstile chain)
>        8        14    13747639      561636    72520256      0      0  0
> 689603 /usr/src/sys/net/route.c:439 (sleep mutex:rtentry)
>        3        20     3907071     2322975    72520256      0      0  0
> 2529589 /usr/src/sys/netinet/ip_input.c:1315 (sleep mutex:rtentry)
>        3        17     3665247     3715117    72520256      0      0  0
> 8425384 /usr/src/sys/netinet/in_rmx.c:114 (sleep mutex:rtentry)

Try disabling net.inet.ip.redirect (sysctl net.inet.ip.redirect=0).
That'll eliminate that in_rmx.c check.

Oh look! The ip_output() path doesn't know about flowtable either.

I'm kind of surprised that the 2-tuple flowtable (ie, only ipv4 and
only ipv6 addresses, not TCP/UDP ports) isn't used in the ip
forwarding case. All ip_rtaddr() is doing is doing a route lookup and
taking a reference to the ifa. It's using that for things like
network/netmask on that interface.

Anyway - yeah, it looks like you've hit lock contention on that
particular setup. You'll likely get a little more throughout out by
disabling redirects for now. The real solution is to make the whole
rtentry locking less stupid and bottleneck-y as well as extending the
flowtable support to include the ip_forward() path.



-a