4 million packets per second: Re: fastforward/routing: a 3 million packet-per-second system?

[John Jasen]

Following up, toggling hw.cxgbe.cong_drop yielded over 4 million packets
per second in testing. This is probably an artifact of the switch I'm
using, and should not be used unless your switch or clients don't respect
ethernet pause.

Turning off ip/icmp redirects yielded a little more, perhaps another 100k
or so.

Increasing the hw.cxgbe.n*xq10g in loader.conf (below) pushed a bottleneck
from t4_transmit into kernel locking, so should be considered for busy
servers.

I removed the kern.sched sysctls, without notable changes to performance.

It looks like the system spends a lot of time in locks, which it looks like
there are patches in HEAD to address this.

There are also patches in HEAD to make the cxgbe drivers work with netmap,
which can also be useful.

more /boot/loader.conf
carp_load="YES"
netmap_load="YES"
cxgbe_load="YES"
net.isr.maxthreads=12
net.isr.bindthreads=1
net.isr.maxqlimit=60480
net.link.ifqmaxlen=90000
# additions in discussions with cxgbe maintainer, Navdeep Parhar (
np at freebsd.org)
hw.cxgbe.toecaps_allowed=0
hw.cxgbe.ntxq10g=48
hw.cxgbe.nrxq10g=16
# end cxgbe recommended additions
# experimenting with cluster size, no effect
hw.cxgbe.largest_rx_cluster=65536
hw.cxgbe.safest_rx_cluster=16384
#cong_drop: increased throughput by ~20%.
# not recommended for use with real switches that respond to tx_pause
hw.cxgbe.cong_drop=1

more /etc/sysctl.conf
net.inet.ip.fastforwarding=1
kern.random.sys.harvest.ethernet=0
kern.random.sys.harvest.point_to_point=0
kern.random.sys.harvest.interrupt=0
hw.intr_storm_threshold=25000000
# not even coming close to this. could be dropped
kern.ipc.maxsockbuf=258291200
net.inet.tcp.recvbuf_max=258291200
net.inet.tcp.sendbuf_max=258291200
#kern.sched.slice=1
#kern.sched.interact=1
net.inet.icmp.log_redirect=0
net.inet.icmp.redirect=0
net.inet.ip.redirect=0
net.inet.icmp.drop_redirect=1

 more /etc/rc.conf
hostname="fb-test"
#ifconfig_igb0="DHCP"
sshd_enable="YES"
#ntpd_enable="YES"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"
# OpenBSD PF options
#pf_enable="YES"
#pflog_enable="YES"
# enable packet forwarding
gateway_enable="YES"
ipv6_gateway_enable="YES"
# enable OpenBSD ftp-procy
ftpproxy_enable="YES"
#aliases used below to help shift clients during testing, and artifact of
#testing other vendor card
ifconfig_cxl0="inet 172.16.3.1 netmask 255.255.255.0 mtu 9000 -lro -tso up"
ifconfig_cxl1="inet 172.16.4.1 netmask 255.255.255.0 mtu 9000 -lro -tso up"
ifconfig_cxl2="inet 172.16.5.1 netmask 255.255.255.0 mtu 9000 -lro -tso up"
ifconfig_cxl3="inet 172.16.6.1 netmask 255.255.255.0 mtu 9000 -lro -tso up"
ifconfig_cxl0_alias0="inet 172.16.7.1 netmask 255.255.255.0"
ifconfig_cxl1_alias0="inet 172.16.8.1 netmask 255.255.255.0"
ifconfig_cxl2_alias0="inet 172.16.1.1 netmask 255.255.255.0"
ifconfig_cxl3_alias0="inet 172.16.2.1 netmask 255.255.255.0"
# ssh interface
ifconfig_igb0="inet 172.30.60.60 netmask 255.255.0.0"


On Tue, Jul 22, 2014 at 11:18 AM, John Jasen <jjasen at gmail.com> wrote:

>
>