[Bug 256707] RPI 4: Packets missing when forwarding through genet0
- Go to: [ bottom of page ] [ top of archives ] [ this month ]
Date: Sat, 19 Jun 2021 05:09:13 UTC
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=256707
Bug ID: 256707
Summary: RPI 4: Packets missing when forwarding through genet0
Product: Base System
Version: 13.0-RELEASE
Hardware: arm64
OS: Any
Status: New
Severity: Affects Only Me
Priority: ---
Component: arm
Assignee: freebsd-arm@FreeBSD.org
Reporter: arroz@me.com
I'm setting up a RPI as a router/WireGuard box, with the following
configuration:
|--------------| |---------|
| | 10.0.1.1 10.0.1.205 | | 10.0.10.1
Public IP --| ISP Router |-----------------------| RPI |----------
Secure LAN
| | genet0 | | ue0
|--------------| |---------|
|
wg0 | 10.0.5.1
|
The PI is connected to the ISP router through genet0, and doing NAT for the
internal LAN (a USB adapter, RealTek chip) and Wireguard.
genet0 behaves well while natd is not running. As soon as natd starts
(in-kernel, through IPFW), the following starts to happen:
- An SSH connection from the 10.0.1.* network to the RPI, through genet0,
starts losing some packets. This is very deterministic, and can be easily
reproduced by running "ifconfig". The response will halt for a bit, while the
client waits for the lost packets to be re-transmitted. Scrolling through a
long manpage also reproduces the problem quite well, for example "man ipfw" and
then keeping the down arrow pressed.
- Wireguard clients simply cannot access anything through the genet0 interface,
forwarding doesn't work at all (but they can access services running on the RPI
itself and machines on the secure LAN).
I've setup an OS from scratch on an external drive, and reduced it to a simple
configuration. As stated above, having IPFW only (without NAT) doesn't affect
the genet0 interface, SSH works reliably. However, as soon as NAT is included
in the configuration, the problems above start happening.
I tried switching the role of both network interfaces, connecting the RPI to
the ISP Router through the USB interface, and things work as expected (no SSH
drops, wireguard clients can access the outside network without issues). I also
have a VM running on an Intel machine with a similar configuration and it works
as expected.
Here's a tcpdump of pressing the return key after typing the "ifconfig"
command. Notice some packets are missing and have to be retransmitted later.
Somehow the packets don't even make to the interface:
-------------
21:34:56.819051 IP 10.0.1.11.55504 > 10.0.1.205.ssh: Flags [P.], seq
3078388568:3078388604, ack 3689303687, win 2048, options [nop,nop,TS val
3670035500 ecr 306290122], length 36
21:34:56.819708 IP 10.0.1.205.ssh > 10.0.1.11.55504: Flags [P.], seq 1:45, ack
36, win 1027, options [nop,nop,TS val 306292760 ecr 3670035500], length 44
21:34:56.821165 IP 10.0.1.11.55504 > 10.0.1.205.ssh: Flags [.], ack 45, win
2047, options [nop,nop,TS val 3670035508 ecr 306292760], length 0
21:34:56.827494 IP 10.0.1.205.ssh > 10.0.1.11.55504: Flags [P.], seq 45:241,
ack 36, win 1027, options [nop,nop,TS val 306292768 ecr 3670035508], length 196
21:34:56.827589 IP 10.0.1.205.ssh > 10.0.1.11.55504: Flags [P.], seq 241:301,
ack 36, win 1027, options [nop,nop,TS val 306292768 ecr 3670035508], length 60
21:34:56.827675 IP 10.0.1.205.ssh > 10.0.1.11.55504: Flags [P.], seq 301:393,
ack 36, win 1027, options [nop,nop,TS val 306292768 ecr 3670035508], length 92
21:34:56.828437 IP 10.0.1.205.ssh > 10.0.1.11.55504: Flags [P.], seq 501:593,
ack 36, win 1027, options [nop,nop,TS val 306292769 ecr 3670035508], length 92
21:34:56.828536 IP 10.0.1.205.ssh > 10.0.1.11.55504: Flags [P.], seq 593:765,
ack 36, win 1027, options [nop,nop,TS val 306292769 ecr 3670035508], length 172
21:34:56.828627 IP 10.0.1.205.ssh > 10.0.1.11.55504: Flags [P.], seq 765:825,
ack 36, win 1027, options [nop,nop,TS val 306292769 ecr 3670035508], length 60
21:34:56.828945 IP 10.0.1.205.ssh > 10.0.1.11.55504: Flags [P.], seq 941:993,
ack 36, win 1027, options [nop,nop,TS val 306292770 ecr 3670035508], length 52
21:34:56.829048 IP 10.0.1.205.ssh > 10.0.1.11.55504: Flags [P.], seq 993:1149,
ack 36, win 1027, options [nop,nop,TS val 306292770 ecr 3670035508], length 156
21:34:56.829137 IP 10.0.1.205.ssh > 10.0.1.11.55504: Flags [P.], seq 1149:1313,
ack 36, win 1027, options [nop,nop,TS val 306292770 ecr 3670035508], length 164
21:34:56.829212 IP 10.0.1.205.ssh > 10.0.1.11.55504: Flags [P.], seq 1313:1405,
ack 36, win 1027, options [nop,nop,TS val 306292770 ecr 3670035508], length 92
21:34:56.829383 IP 10.0.1.11.55504 > 10.0.1.205.ssh: Flags [.], ack 241, win
2044, options [nop,nop,TS val 3670035515 ecr 306292768], length 0
21:34:56.830175 IP 10.0.1.11.55504 > 10.0.1.205.ssh: Flags [.], ack 301, win
2047, options [nop,nop,TS val 3670035515 ecr 306292768], length 0
21:34:56.830217 IP 10.0.1.11.55504 > 10.0.1.205.ssh: Flags [.], ack 393, win
2046, options [nop,nop,TS val 3670035516 ecr 306292768], length 0
21:34:56.831738 IP 10.0.1.205.ssh > 10.0.1.11.55504: Flags [P.], seq 1405:1497,
ack 36, win 1027, options [nop,nop,TS val 306292773 ecr 3670035516], length 92
21:34:56.831841 IP 10.0.1.205.ssh > 10.0.1.11.55504: Flags [P.], seq 1497:1589,
ack 36, win 1027, options [nop,nop,TS val 306292773 ecr 3670035516], length 92
21:34:56.831940 IP 10.0.1.205.ssh > 10.0.1.11.55504: Flags [P.], seq 1589:1777,
ack 36, win 1027, options [nop,nop,TS val 306292773 ecr 3670035516], length 188
21:34:56.832018 IP 10.0.1.11.55504 > 10.0.1.205.ssh: Flags [.], ack 393, win
2048, options [nop,nop,TS val 3670035517 ecr 306292768,nop,nop,sack 1
{501:593}], length 0
21:34:56.832027 IP 10.0.1.205.ssh > 10.0.1.11.55504: Flags [P.], seq 1777:1829,
ack 36, win 1027, options [nop,nop,TS val 306292773 ecr 3670035516], length 52
21:34:56.833431 IP 10.0.1.11.55504 > 10.0.1.205.ssh: Flags [.], ack 393, win
2048, options [nop,nop,TS val 3670035518 ecr 306292768,nop,nop,sack 1
{501:765}], length 0
21:34:56.833985 IP 10.0.1.205.ssh > 10.0.1.11.55504: Flags [P.], seq 1937:2005,
ack 36, win 1027, options [nop,nop,TS val 306292775 ecr 3670035518], length 68
21:34:56.834635 IP 10.0.1.11.55504 > 10.0.1.205.ssh: Flags [.], ack 393, win
2048, options [nop,nop,TS val 3670035518 ecr 306292768,nop,nop,sack 1
{501:825}], length 0
21:34:56.834705 IP 10.0.1.11.55504 > 10.0.1.205.ssh: Flags [.], ack 393, win
2048, options [nop,nop,TS val 3670035519 ecr 306292768,nop,nop,sack 2
{941:993}{501:825}], length 0
21:34:56.835667 IP 10.0.1.11.55504 > 10.0.1.205.ssh: Flags [.], ack 393, win
2048, options [nop,nop,TS val 3670035519 ecr 306292768,nop,nop,sack 2
{941:1149}{501:825}], length 0
21:34:56.836347 IP 10.0.1.11.55504 > 10.0.1.205.ssh: Flags [.], ack 393, win
2048, options [nop,nop,TS val 3670035520 ecr 306292768,nop,nop,sack 2
{941:1313}{501:825}], length 0
21:34:56.837672 IP 10.0.1.11.55504 > 10.0.1.205.ssh: Flags [.], ack 393, win
2048, options [nop,nop,TS val 3670035521 ecr 306292768,nop,nop,sack 2
{941:1405}{501:825}], length 0
21:34:56.846477 IP 10.0.1.11.55504 > 10.0.1.205.ssh: Flags [.], ack 393, win
2048, options [nop,nop,TS val 3670035523 ecr 306292768,nop,nop,sack 2
{941:1497}{501:825}], length 0
21:34:56.846515 IP 10.0.1.11.55504 > 10.0.1.205.ssh: Flags [.], ack 393, win
2048, options [nop,nop,TS val 3670035523 ecr 306292768,nop,nop,sack 2
{941:1589}{501:825}], length 0
21:34:56.846533 IP 10.0.1.11.55504 > 10.0.1.205.ssh: Flags [.], ack 393, win
2048, options [nop,nop,TS val 3670035523 ecr 306292768,nop,nop,sack 2
{941:1777}{501:825}], length 0
21:34:56.846550 IP 10.0.1.11.55504 > 10.0.1.205.ssh: Flags [.], ack 393, win
2048, options [nop,nop,TS val 3670035524 ecr 306292768,nop,nop,sack 2
{941:1829}{501:825}], length 0
21:34:56.846702 IP 10.0.1.11.55504 > 10.0.1.205.ssh: Flags [.], ack 393, win
2048, options [nop,nop,TS val 3670035526 ecr 306292768,nop,nop,sack 3
{1937:2005}{941:1829}{501:825}], length 0
21:34:57.083315 IP 10.0.1.205.ssh > 10.0.1.11.55504: Flags [.], seq 393:1841,
ack 36, win 1027, options [nop,nop,TS val 306293025 ecr 3670035526], length
1448
21:34:57.105699 IP 10.0.1.11.55504 > 10.0.1.205.ssh: Flags [.], ack 1841, win
2025, options [nop,nop,TS val 3670035784 ecr 306293025,nop,nop,sack 2
{501:1713}{1937:2005}], length 0
21:34:57.105754 IP 10.0.1.205.ssh > 10.0.1.11.55504: Flags [P.], seq 1841:2005,
ack 36, win 1027, options [nop,nop,TS val 306293047 ecr 3670035784], length 164
21:34:57.107311 IP 10.0.1.11.55504 > 10.0.1.205.ssh: Flags [.], ack 2005, win
2045, options [nop,nop,TS val 3670035785 ecr 306293047,nop,nop,sack 1
{1937:2005}], length 0
-------------
A tcpdump of the wireguard broken forwarding shows the packets arrive to the
wg0 interface but never leave through the genet0 one.
tso is disabled:
-------------
% sysctl net.inet.tcp.tso
net.inet.tcp.tso: 0
-------------
Relevant rc.conf lines:
-------------
gateway_enable="YES"
firewall_enable="YES"
firewall_nat_enable="YES"
-------------
ipfw config, pretty much directly from the handbook:
-------------
#!/bin/sh
ipfw -q -f flush
cmd="ipfw -q add"
skip="skipto 1000"
pif=genet0
ks="keep-state"
ipfw disable one_pass
ipfw -q nat 1 config if $pif same_ports unreg_only reset
$cmd 004 allow all from any to any via ue0 # exclude LAN traffic
$cmd 005 allow all from any to any via wg0
$cmd 010 allow all from any to any via lo0 # exclude loopback traffic
$cmd 099 reass all from any to any in # reassemble inbound packets
$cmd 100 nat 1 ip from any to any in via $pif # NAT any inbound packets
# Allow the packet through if it has an existing entry in the dynamic rules
table
$cmd 101 check-state
# Authorized outbound packets
$cmd 120 $skip udp from any to any out via $pif $ks
$cmd 125 $skip tcp from any to any out via $pif setup $ks
$cmd 130 $skip icmp from any to any out via $pif $ks
# Allow SSH and Wireguard
$cmd 200 $skip tcp from any to me 22 setup $ks
$cmd 201 $skip udp from any to me 51820 in via $pif $ks
$cmd 999 deny log all from any to any
$cmd 1000 nat 1 ip from any to any out via $pif
$cmd 1001 allow ip from any to any
-------------
Speculation: I'm assuming this is the IPv4 version of bug #254060, fixed in
https://cgit.freebsd.org/src/commit/sys/arm64/broadcom/genet/if_genet.c?id=2bdcf6237744b2d9d9707d623660d33931daeb52
--
You are receiving this mail because:
You are the assignee for the bug.