TCP BBR in 13.0-RC2 not playing well with pf firewall
Ashish SHUKLA
ashish at FreeBSD.org
Sat Mar 13 10:25:58 UTC 2021
[sorry for some reason, the previous message is showing as empty in archives :(]
Hi,
I just wanted to try TCP BBR functionality available in FreeBSD 13.0-RC2 and noticed that it’s not usable, if FreeBSD pf is enabled.
I’m using following pf rules:
===============
block drop all
block drop in quick on em0 from <badguys> to any
block drop in quick on em0 from <sshguard> to any label "ssh bruteforce"
pass in on em0 proto udp from any to any port = domain keep state
pass in on em0 proto udp from any to any port = 2015 keep state
pass in on em0 proto udp from any to any port = 5001 keep state
pass in on em0 proto udp from any to any port = 6881 keep state
pass in on em0 proto udp from any to any port = 51234 keep state
pass in on em0 proto udp from any to any port 54000:54322 keep state
pass in on em0 proto udp from any to any port 60000:61000 keep state
pass in on em0 proto tcp from any to any port = ssh flags S/SA keep state
pass in on em0 proto tcp from any to any port = xmpp-client flags S/SA keep state
pass in on em0 proto tcp from any to any port = xmpp-server flags S/SA keep state
pass in on em0 proto tcp from any to any port 54000:54322 flags S/SA keep state
pass in on em0 proto icmp all keep state
pass in on em0 proto ipv6-icmp all keep state
pass out on em0 all flags S/SA keep state
===============
Following is how I tried to reproduce (same with IPv6):
===============
% wget -4 -O /dev/null https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.11.3.tar.xz
--2021-03-13 09:26:23-- https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.11.3.tar.xz
Resolving cdn.kernel.org (cdn.kernel.org)... 151.101.113.176
Connecting to cdn.kernel.org (cdn.kernel.org)|151.101.113.176|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 117629864 (112M) [application/x-xz]
Saving to: ‘/dev/null’
/dev/null 0%[ ] 137.44K --.-KB/s in 10s
2021-03-13 09:26:33 (13.8 KB/s) - Read error at byte 140737/117629864 (Network is down). Retrying.
--2021-03-13 09:26:34-- (try: 2) https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.11.3.tar.xz
Connecting to cdn.kernel.org (cdn.kernel.org)|151.101.113.176|:443... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 117629864 (112M), 117489127 (112M) remaining [application/x-xz]
Saving to: ‘/dev/null’
/dev/null 0%[ ] 243.21K --.-KB/s in 10s
2021-03-13 09:26:44 (10.6 KB/s) - Read error at byte 249046/117629864 (Network is down). Retrying.
===============
A copy of tcpdump output (tcpdump -i em0 -nv port 443 and host 151.101.113.176) is uploaded[1].
Since TCP BBR requires custom kernel, following is the diff:
===============
--- GENERIC 2021-02-23 10:55:03.397376000 +0000
+++ MYKERNEL 2021-02-23 10:58:39.442981000 +0000
@@ -19,7 +19,7 @@
# $FreeBSD$
cpu HAMMER
-ident GENERIC
+ident MYKERNEL
makeoptions DEBUG=-g # Build kernel with gdb(1) debug symbols
makeoptions WITH_CTF=1 # Run ctfconvert(1) for DTrace support
@@ -382,3 +382,6 @@
options HID_DEBUG # enable debug msgs
device hid # Generic HID support
options IICHID_SAMPLING # Workaround missing GPIO INTR support
+
+options ROUTETABLES=12
+options TCPHPTS
===============
And following src.conf:
===============
WITH_EXTRA_TCP_STACKS= yes
===============
sysctl.conf(5) is empty, except following sysctl knob set to switch to BBR stack:
===============
net.inet.tcp.functions_default=bbr
===============
Following are the contents of loader.conf(5):
===============
kern.geom.label.disk_ident.enable="0"
kern.geom.label.gptid.enable="0"
opensolaris_load="YES"
zfs_load="YES"
debug.acpi.disabled="thermal"
coretemp_load=YES
vm.pmap.pti=0
cpu_microcode_load="YES"
cpu_microcode_name="/boot/firmware/intel-ucode.bin"
aesni_load=YES
nullfs_load="YES"
vfs.zfs.prefetch_disable=1
vfs.zfs.write_limit_override=1073741824
vfs.zfs.min_auto_ashift=12
vfs.zfs.arc_max="17179869184"
vfs.zfs.compressed_arc_enabled=1
===============
Following is the output of kldstat:
===============
Id Refs Address Size Name
1 33 0xffffffff80200000 1f12e50 kernel
2 1 0xffffffff82113000 b7b8 opensolaris.ko
3 1 0xffffffff8211f000 9ac0 nullfs.ko
4 1 0xffffffff8249d000 67fbc8 zfs.ko
5 1 0xffffffff82b1d000 4128 coretemp.ko
6 1 0xffffffff82d20000 3378 acpi_wmi.ko
7 1 0xffffffff82d24000 3250 ichsmb.ko
8 1 0xffffffff82d28000 2180 smbus.ko
9 1 0xffffffff82d2b000 2110 pchtherm.ko
10 1 0xffffffff82d2e000 2a08 mac_ntpd.ko
11 1 0xffffffff82d31000 29bc8 tcp_bbr.ko
12 1 0xffffffff82d5b000 36f70 pf.ko
===============
Host is running kernel from git revision “13c22f7495305f5b92874128b088ab47d9512c20”, while rest of the userland is from "freebsd-update upgrade -r 13.0-RC2”.
After disabling pf firewall, it becomes usable again:
===============
--2021-03-13 09:42:50-- https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.11.3.tar.xz
Resolving cdn.kernel.org (cdn.kernel.org)... 151.101.113.176
Connecting to cdn.kernel.org (cdn.kernel.org)|151.101.113.176|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 117629864 (112M) [application/x-xz]
Saving to: ‘/dev/null’
/dev/null 100%[======================================================================================================>] 112.18M 20.2MB/s in 6.9s
2021-03-13 09:42:57 (16.3 MB/s) - ‘/dev/null’ saved [117629864/117629864]
===============
Please let me know if I overlooked something with TCP BBR configuration.
References:
[1] https://people.freebsd.org/~ashish/tcpdump-cdn-kernel-org-20210313.txt
Thanks!
--
Ashish | GPG: F682 CDCC 39DC 0FEA E116 20B6 C746 CFA9 E74F A4B0
“Sometimes even to live is an act of courage.” (Seneca)
More information about the freebsd-stable
mailing list