fastforward/routing: a 3 million packet-per-second system?

[John Jasen]

Based on advice I received, I've collected lock profile debugging output,
and pmcannotate'd data from the system while it was processing about 3
million packets/second.

Combined, the files are about 325k in size, so I'll submit highlights here.
I can provide the raw files to interested parties privately.

pmcannotate summary output:

grep ^Profile pmcannotate.20140725
Profile trace for function: __rw_rlock() [17.04%]
Profile trace for function: __mtx_unlock_flags() [9.10%]
Profile trace for function: _rw_runlock_cookie() [7.67%]
Profile trace for function: sched_idletd() [5.73%]
Profile trace for function: memcpy() [5.64%]
Profile trace for function: bcopy() [5.04%]
Profile trace for function: bcmp() [5.01%]
Profile trace for function: __mtx_lock_flags() [3.66%]
Profile trace for function: t4_eth_tx() [3.25%]
Profile trace for function: lock_profile_release_lock() [2.73%]
Profile trace for function: ip_fastforward() [2.68%]
Profile trace for function: ether_output() [2.50%]
Profile trace for function: get_scatter_segment() [1.75%]
Profile trace for function: rn_match() [1.74%]
Profile trace for function: _mtx_lock_spin_cookie() [1.53%]
Profile trace for function: lock_profile_obtain_lock_success() [1.49%]
Profile trace for function: cxgbe_transmit() [1.37%]
Profile trace for function: uma_zalloc_arg() [1.31%]
Profile trace for function: bzero() [1.30%]
Profile trace for function: service_iq() [1.26%]
Profile trace for function: ether_nh_input() [1.23%]
Profile trace for function: __mtx_lock_sleep() [1.19%]
Profile trace for function: arpresolve() [1.07%]
Profile trace for function: uma_zfree_arg() [0.95%]
Profile trace for function: reclaim_tx_descs() [0.87%]
Profile trace for function: _mtx_trylock_flags_() [0.80%]
Profile trace for function: bounce_bus_dmamap_load_buffer() [0.72%]
Profile trace for function: ether_demux() [0.64%]
Profile trace for function: mb_ctor_mbuf() [0.63%]
Profile trace for function: rtalloc1_fib() [0.54%]

sysctl debug.lock.prof.stats summary: (some of the highest hit counts,
especially in cnt_hold:

       7        17     3271389     2258698    63952832      0      0  0
6761237 /usr/src/sys/netinet/in_rmx.c:114 (sleep mutex:rtentry)

       7        13    11543138      470545    63952832      0      0  0
815777 /usr/src/sys/net/route.c:439 (sleep mutex:rtentry)

       6        15     3943582     1545195    63952779      0      0  0
3439254 /usr/src/sys/netinet/ip_fastfwd.c:593 (sleep mutex:rtentry











On Tue, Jul 22, 2014 at 11:18 AM, John Jasen <jjasen at gmail.com> wrote:

> Feedback and/or tips and tricks more than welcome.
>

<snip>