High UDP loss on 11.2-RELEASE-p4

[Greg Rivers]

I'm seeing considerable packet loss running recursive DNS (BIND 9.12.2) on 11.2. This is on server grade hardware (HP DL380G8) with an Intel 10Gb NIC (ix).

I spun up iperf3; the results were disappointing:

$ iperf3 --get-server-output -c r5 -u -b 1g -R   
Connecting to host r5, port 5201
Reverse mode, remote host r5 is sending
[  5] local xxx.xxx.242.11 port 56921 connected to xxx.xxx.235.238 port 5201
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-1.00   sec  77.0 MBytes   645 Mbits/sec  0.003 ms  32835/88105 (37%)  
[  5]   1.00-2.00   sec  72.2 MBytes   605 Mbits/sec  0.002 ms  33813/85638 (39%)  
[  5]   2.00-3.00   sec  71.3 MBytes   599 Mbits/sec  0.003 ms  34291/85534 (40%)  
[  5]   3.00-4.00   sec  70.0 MBytes   587 Mbits/sec  0.012 ms  35459/85757 (41%)  
[  5]   4.00-5.00   sec  71.9 MBytes   603 Mbits/sec  0.002 ms  34053/85688 (40%)  
[  5]   5.00-6.00   sec  72.7 MBytes   609 Mbits/sec  0.004 ms  33354/85555 (39%)  
[  5]   6.00-7.00   sec  71.7 MBytes   602 Mbits/sec  0.002 ms  33984/85472 (40%)  
[  5]   7.00-8.00   sec  71.0 MBytes   596 Mbits/sec  0.002 ms  34618/85623 (40%)  
[  5]   8.00-9.00   sec  70.8 MBytes   594 Mbits/sec  0.002 ms  34773/85630 (41%)  
[  5]   9.00-10.00  sec  71.3 MBytes   598 Mbits/sec  0.002 ms  34593/85784 (40%)  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  1.17 GBytes  1.00 Gbits/sec  0.000 ms  0/858786 (0%)  sender
[  5]   0.00-10.00  sec   720 MBytes   604 Mbits/sec  0.002 ms  341773/858786 (40%)  receiver

Server output:
-----------------------------------------------------------
Accepted connection from xxx.xxx.242.11, port 54163
[  5] local xxx.xxx.235.238 port 5201 connected to xxx.xxx.242.11 port 56921
[ ID] Interval           Transfer     Bandwidth       Total Datagrams
[  5]   0.00-1.00   sec   111 MBytes   930 Mbits/sec  79657  
[  5]   1.00-2.00   sec   119 MBytes   999 Mbits/sec  85536  
[  5]   2.00-3.00   sec   119 MBytes  1.00 Gbits/sec  85625  
[  5]   3.00-4.00   sec   119 MBytes   999 Mbits/sec  85541  
[  5]   4.00-5.00   sec   119 MBytes  1.00 Gbits/sec  85667  
[  5]   5.00-6.00   sec   119 MBytes  1000 Mbits/sec  85592  
[  5]   6.00-7.00   sec   119 MBytes  1.00 Gbits/sec  85720  
[  5]   7.00-8.00   sec   119 MBytes   998 Mbits/sec  85457  
[  5]   8.00-9.00   sec   119 MBytes  1.00 Gbits/sec  85622  
[  5]   9.00-10.00  sec   119 MBytes  1000 Mbits/sec  85603  
[  5]  10.00-10.10  sec  12.2 MBytes  1.02 Gbits/sec  8767  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  5]   0.00-10.10  sec  1.17 GBytes   993 Mbits/sec  0.000 ms  0/858787 (0%)


40% loss at 1Gb.

I'm running with the following tunes, which did improve DNS service when I applied them:
dev.ix.0.tx_processing_limit: -1
hw.ix.flow_control: 0
hw.ix.num_queues: 8
hw.ix.rx_process_limit: -1
hw.ix.rxd: 2048
hw.ix.tx_process_limit: -1
hw.ix.txd: 2048
kern.ipc.maxsockbuf: 16777216
kern.random.harvest.mask: 351
machdep.hyperthreading_allowed: 0
net.inet.ip.intr_queue_maxlen: 2048
net.inet.raw.maxdgram: 16384
net.inet.raw.recvspace: 16384
net.isr.bindthreads: 1
net.isr.defaultqlimit: 2048
net.isr.maxthreads: 16
net.link.ifqmaxlen: 4096
net.local.stream.recvspace: 16384
net.local.stream.sendspace: 16384
net.route.netisr_maxqlen: 2048

Are there any known problems with 11.2 and UDP? Can anyone else reproduce this? Any insight or suggestions?

-- 
Greg