Issue with BGP router / high interrupt / Chelsio / FreeBSD 12.1

[Rudy Rucker]

On 2/14/20 2:58 PM, BulkMailForRudy wrote:
> pmcstat -S inst_retired.any -Tw1 

I had time to run pmcstat during a larger TX load on Chelsio T6 card.

I'm loosing a hope on using FreeBSD as one of my edge routers outside of 
10Gbps ports.


*Start (been running for 4 days), cc1: RX 2.3Gbps / 130Mbps*

%SAMP IMAGE      FUNCTION             CALLERS
  20.4 kernel     sched_idletd         fork_exit
  14.1 kernel     cpu_search_highest   cpu_search_highest:11.9 
sched_switch:1.3 sched_idletd:0.9
   5.5 kernel     cpu_search_lowest    cpu_search_lowest:5.0 
sched_pickcpu:0.5
   4.6 kernel     eth_tx               drain_ring
   3.7 kernel     rn_match             fib4_lookup_nh_basic
   2.0 kernel     epoch_enter_preempt  arpresolve
   2.0 kernel     mac_ifnet_check_tran ether_output
   1.9 kernel     lock_delay           __mtx_lock_sleep
   1.7 kernel     service_iq_fl        t4_intr
   1.6 kernel     ip_tryforward        ip_input
   1.5 kernel     get_scatter_segment  service_iq_fl
   1.5 kernel     mac_ifnet_create_mbu ether_nh_input
   1.4 kernel     parse_pkt            cxgbe_transmit
   1.3 kernel     netisr_dispatch_src  ether_demux:0.7 ether_input:0.6
   1.3 kernel     cxgbe_transmit       ether_output_frame
   1.2 kernel     _rm_rlock            in_localip
   1.2 kernel     mp_ring_enqueue      cxgbe_transmit
   1.2 kernel     ether_output         ip_tryforward

*5 minutes in sending more traffic out cc1, cc1: RX 2.3Gbps  / TX 1.7Gbps*


%SAMP IMAGE      FUNCTION             CALLERS
  17.0 kernel     sched_idletd         fork_exit
  15.3 kernel     cpu_search_highest   cpu_search_highest:13.0 
sched_switch:1.3 sched_idletd:1.0
   6.9 kernel     cpu_search_lowest    cpu_search_lowest:6.3 
sched_pickcpu:0.6
   4.8 kernel     eth_tx               drain_ring
   3.7 kernel     rn_match             fib4_lookup_nh_basic
   2.7 kernel     lock_delay           __mtx_lock_sleep
   2.0 kernel     mac_ifnet_check_tran ether_output
   1.9 kernel     epoch_enter_preempt  arpresolve
   1.8 kernel     service_iq_fl        t4_intr
   1.6 kernel     mac_ifnet_create_mbu ether_nh_input
   1.6 kernel     ip_tryforward        ip_input


*10 minutes in, attempting to make cc1 referred outbound for whole 
network (~ 5Gbps), and the box starts dropping packets:*

PMC: [inst_retired.any] Samples: 1019841 (100.0%) , 19068 unresolved

%SAMP IMAGE      FUNCTION             CALLERS
  24.8 kernel     cpu_search_highest   cpu_search_highest:22.7 
sched_idletd:1.6
   6.2 kernel     lock_delay           __mtx_lock_sleep
   6.0 kernel     rn_match             fib4_lookup_nh_basic
   5.7 kernel     eth_tx               drain_ring
   2.7 kernel     sched_idletd         fork_exit
   2.6 kernel     tdq_move             sched_idletd
   2.5 kernel     mac_ifnet_check_tran ether_output
   2.1 kernel     mac_ifnet_create_mbu ether_nh_input
   2.0 kernel     epoch_enter_preempt  arpresolve

*Reverting to ~ 150Mbps TX on  cc1*

PMC: [inst_retired.any] Samples: 136053 (100.0%) , 2767 unresolved

%SAMP IMAGE      FUNCTION             CALLERS
  27.1 kernel     sched_idletd         fork_exit
  14.1 kernel     cpu_search_highest   cpu_search_highest:11.5 
sched_switch:1.6 sched_idletd:0.9
   4.4 kernel     cpu_search_lowest    cpu_search_lowest
   4.1 kernel     eth_tx               drain_ring
   3.4 kernel     rn_match             fib4_lookup_nh_basic
   1.8 kernel     epoch_enter_preempt  arpresolve
   1.8 kernel     mac_ifnet_check_tran ether_output
   1.6 kernel     service_iq_fl        t4_intr
   1.4 kernel     ip_tryforward        ip_input
   1.4 kernel     get_scatter_segment  service_iq_fl
   1.3 kernel     mac_ifnet_create_mbu ether_nh_input
   1.3 kernel     sched_switch         mi_switch
   1.3 kernel     parse_pkt            cxgbe_transmit
   1.2 kernel     _rm_rlock            in_localip