From nobody Sat Jun 19 05:09:13 2021 X-Original-To: freebsd-arm@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 991785D3EB0 for ; Sat, 19 Jun 2021 05:09:13 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4G6P3F3Z87z4dp5 for ; Sat, 19 Jun 2021 05:09:13 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 5CE87105BD for ; Sat, 19 Jun 2021 05:09:13 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 15J59Dwg012368 for ; Sat, 19 Jun 2021 05:09:13 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 15J59DD5012367 for freebsd-arm@FreeBSD.org; Sat, 19 Jun 2021 05:09:13 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: freebsd-arm@FreeBSD.org Subject: [Bug 256707] RPI 4: Packets missing when forwarding through genet0 Date: Sat, 19 Jun 2021 05:09:13 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: arm X-Bugzilla-Version: 13.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: arroz@me.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-arm@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Porting FreeBSD to ARM processors List-Archive: https://lists.freebsd.org/archives/freebsd-arm List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arm@freebsd.org MIME-Version: 1.0 X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D256707 Bug ID: 256707 Summary: RPI 4: Packets missing when forwarding through genet0 Product: Base System Version: 13.0-RELEASE Hardware: arm64 OS: Any Status: New Severity: Affects Only Me Priority: --- Component: arm Assignee: freebsd-arm@FreeBSD.org Reporter: arroz@me.com I'm setting up a RPI as a router/WireGuard box, with the following configuration: |--------------| |---------| | | 10.0.1.1 10.0.1.205 | | 10.0.10.1 Public IP --| ISP Router |-----------------------| RPI |---------- Secure LAN | | genet0 | | ue0 |--------------| |---------| | wg0 | 10.0.5.1 | The PI is connected to the ISP router through genet0, and doing NAT for the internal LAN (a USB adapter, RealTek chip) and Wireguard. genet0 behaves well while natd is not running. As soon as natd starts (in-kernel, through IPFW), the following starts to happen: - An SSH connection from the 10.0.1.* network to the RPI, through genet0, starts losing some packets. This is very deterministic, and can be easily reproduced by running "ifconfig". The response will halt for a bit, while t= he client waits for the lost packets to be re-transmitted. Scrolling through a long manpage also reproduces the problem quite well, for example "man ipfw"= and then keeping the down arrow pressed. - Wireguard clients simply cannot access anything through the genet0 interf= ace, forwarding doesn't work at all (but they can access services running on the= RPI itself and machines on the secure LAN). I've setup an OS from scratch on an external drive, and reduced it to a sim= ple configuration. As stated above, having IPFW only (without NAT) doesn't affe= ct the genet0 interface, SSH works reliably. However, as soon as NAT is includ= ed in the configuration, the problems above start happening. I tried switching the role of both network interfaces, connecting the RPI to the ISP Router through the USB interface, and things work as expected (no S= SH drops, wireguard clients can access the outside network without issues). I = also have a VM running on an Intel machine with a similar configuration and it w= orks as expected. Here's a tcpdump of pressing the return key after typing the "ifconfig" command. Notice some packets are missing and have to be retransmitted later. Somehow the packets don't even make to the interface: ------------- 21:34:56.819051 IP 10.0.1.11.55504 > 10.0.1.205.ssh: Flags [P.], seq 3078388568:3078388604, ack 3689303687, win 2048, options [nop,nop,TS val 3670035500 ecr 306290122], length 36 21:34:56.819708 IP 10.0.1.205.ssh > 10.0.1.11.55504: Flags [P.], seq 1:45, = ack 36, win 1027, options [nop,nop,TS val 306292760 ecr 3670035500], length 44 21:34:56.821165 IP 10.0.1.11.55504 > 10.0.1.205.ssh: Flags [.], ack 45, win 2047, options [nop,nop,TS val 3670035508 ecr 306292760], length 0 21:34:56.827494 IP 10.0.1.205.ssh > 10.0.1.11.55504: Flags [P.], seq 45:241, ack 36, win 1027, options [nop,nop,TS val 306292768 ecr 3670035508], length= 196 21:34:56.827589 IP 10.0.1.205.ssh > 10.0.1.11.55504: Flags [P.], seq 241:30= 1, ack 36, win 1027, options [nop,nop,TS val 306292768 ecr 3670035508], length= 60 21:34:56.827675 IP 10.0.1.205.ssh > 10.0.1.11.55504: Flags [P.], seq 301:39= 3, ack 36, win 1027, options [nop,nop,TS val 306292768 ecr 3670035508], length= 92 21:34:56.828437 IP 10.0.1.205.ssh > 10.0.1.11.55504: Flags [P.], seq 501:59= 3, ack 36, win 1027, options [nop,nop,TS val 306292769 ecr 3670035508], length= 92 21:34:56.828536 IP 10.0.1.205.ssh > 10.0.1.11.55504: Flags [P.], seq 593:76= 5, ack 36, win 1027, options [nop,nop,TS val 306292769 ecr 3670035508], length= 172 21:34:56.828627 IP 10.0.1.205.ssh > 10.0.1.11.55504: Flags [P.], seq 765:82= 5, ack 36, win 1027, options [nop,nop,TS val 306292769 ecr 3670035508], length= 60 21:34:56.828945 IP 10.0.1.205.ssh > 10.0.1.11.55504: Flags [P.], seq 941:99= 3, ack 36, win 1027, options [nop,nop,TS val 306292770 ecr 3670035508], length= 52 21:34:56.829048 IP 10.0.1.205.ssh > 10.0.1.11.55504: Flags [P.], seq 993:11= 49, ack 36, win 1027, options [nop,nop,TS val 306292770 ecr 3670035508], length= 156 21:34:56.829137 IP 10.0.1.205.ssh > 10.0.1.11.55504: Flags [P.], seq 1149:1= 313, ack 36, win 1027, options [nop,nop,TS val 306292770 ecr 3670035508], length= 164 21:34:56.829212 IP 10.0.1.205.ssh > 10.0.1.11.55504: Flags [P.], seq 1313:1= 405, ack 36, win 1027, options [nop,nop,TS val 306292770 ecr 3670035508], length= 92 21:34:56.829383 IP 10.0.1.11.55504 > 10.0.1.205.ssh: Flags [.], ack 241, win 2044, options [nop,nop,TS val 3670035515 ecr 306292768], length 0 21:34:56.830175 IP 10.0.1.11.55504 > 10.0.1.205.ssh: Flags [.], ack 301, win 2047, options [nop,nop,TS val 3670035515 ecr 306292768], length 0 21:34:56.830217 IP 10.0.1.11.55504 > 10.0.1.205.ssh: Flags [.], ack 393, win 2046, options [nop,nop,TS val 3670035516 ecr 306292768], length 0 21:34:56.831738 IP 10.0.1.205.ssh > 10.0.1.11.55504: Flags [P.], seq 1405:1= 497, ack 36, win 1027, options [nop,nop,TS val 306292773 ecr 3670035516], length= 92 21:34:56.831841 IP 10.0.1.205.ssh > 10.0.1.11.55504: Flags [P.], seq 1497:1= 589, ack 36, win 1027, options [nop,nop,TS val 306292773 ecr 3670035516], length= 92 21:34:56.831940 IP 10.0.1.205.ssh > 10.0.1.11.55504: Flags [P.], seq 1589:1= 777, ack 36, win 1027, options [nop,nop,TS val 306292773 ecr 3670035516], length= 188 21:34:56.832018 IP 10.0.1.11.55504 > 10.0.1.205.ssh: Flags [.], ack 393, win 2048, options [nop,nop,TS val 3670035517 ecr 306292768,nop,nop,sack 1 {501:593}], length 0 21:34:56.832027 IP 10.0.1.205.ssh > 10.0.1.11.55504: Flags [P.], seq 1777:1= 829, ack 36, win 1027, options [nop,nop,TS val 306292773 ecr 3670035516], length= 52 21:34:56.833431 IP 10.0.1.11.55504 > 10.0.1.205.ssh: Flags [.], ack 393, win 2048, options [nop,nop,TS val 3670035518 ecr 306292768,nop,nop,sack 1 {501:765}], length 0 21:34:56.833985 IP 10.0.1.205.ssh > 10.0.1.11.55504: Flags [P.], seq 1937:2= 005, ack 36, win 1027, options [nop,nop,TS val 306292775 ecr 3670035518], length= 68 21:34:56.834635 IP 10.0.1.11.55504 > 10.0.1.205.ssh: Flags [.], ack 393, win 2048, options [nop,nop,TS val 3670035518 ecr 306292768,nop,nop,sack 1 {501:825}], length 0 21:34:56.834705 IP 10.0.1.11.55504 > 10.0.1.205.ssh: Flags [.], ack 393, win 2048, options [nop,nop,TS val 3670035519 ecr 306292768,nop,nop,sack 2 {941:993}{501:825}], length 0 21:34:56.835667 IP 10.0.1.11.55504 > 10.0.1.205.ssh: Flags [.], ack 393, win 2048, options [nop,nop,TS val 3670035519 ecr 306292768,nop,nop,sack 2 {941:1149}{501:825}], length 0 21:34:56.836347 IP 10.0.1.11.55504 > 10.0.1.205.ssh: Flags [.], ack 393, win 2048, options [nop,nop,TS val 3670035520 ecr 306292768,nop,nop,sack 2 {941:1313}{501:825}], length 0 21:34:56.837672 IP 10.0.1.11.55504 > 10.0.1.205.ssh: Flags [.], ack 393, win 2048, options [nop,nop,TS val 3670035521 ecr 306292768,nop,nop,sack 2 {941:1405}{501:825}], length 0 21:34:56.846477 IP 10.0.1.11.55504 > 10.0.1.205.ssh: Flags [.], ack 393, win 2048, options [nop,nop,TS val 3670035523 ecr 306292768,nop,nop,sack 2 {941:1497}{501:825}], length 0 21:34:56.846515 IP 10.0.1.11.55504 > 10.0.1.205.ssh: Flags [.], ack 393, win 2048, options [nop,nop,TS val 3670035523 ecr 306292768,nop,nop,sack 2 {941:1589}{501:825}], length 0 21:34:56.846533 IP 10.0.1.11.55504 > 10.0.1.205.ssh: Flags [.], ack 393, win 2048, options [nop,nop,TS val 3670035523 ecr 306292768,nop,nop,sack 2 {941:1777}{501:825}], length 0 21:34:56.846550 IP 10.0.1.11.55504 > 10.0.1.205.ssh: Flags [.], ack 393, win 2048, options [nop,nop,TS val 3670035524 ecr 306292768,nop,nop,sack 2 {941:1829}{501:825}], length 0 21:34:56.846702 IP 10.0.1.11.55504 > 10.0.1.205.ssh: Flags [.], ack 393, win 2048, options [nop,nop,TS val 3670035526 ecr 306292768,nop,nop,sack 3 {1937:2005}{941:1829}{501:825}], length 0 21:34:57.083315 IP 10.0.1.205.ssh > 10.0.1.11.55504: Flags [.], seq 393:184= 1, ack 36, win 1027, options [nop,nop,TS val 306293025 ecr 3670035526], length 1448 21:34:57.105699 IP 10.0.1.11.55504 > 10.0.1.205.ssh: Flags [.], ack 1841, w= in 2025, options [nop,nop,TS val 3670035784 ecr 306293025,nop,nop,sack 2 {501:1713}{1937:2005}], length 0 21:34:57.105754 IP 10.0.1.205.ssh > 10.0.1.11.55504: Flags [P.], seq 1841:2= 005, ack 36, win 1027, options [nop,nop,TS val 306293047 ecr 3670035784], length= 164 21:34:57.107311 IP 10.0.1.11.55504 > 10.0.1.205.ssh: Flags [.], ack 2005, w= in 2045, options [nop,nop,TS val 3670035785 ecr 306293047,nop,nop,sack 1 {1937:2005}], length 0 ------------- A tcpdump of the wireguard broken forwarding shows the packets arrive to the wg0 interface but never leave through the genet0 one. tso is disabled: ------------- % sysctl net.inet.tcp.tso net.inet.tcp.tso: 0 ------------- Relevant rc.conf lines: ------------- gateway_enable=3D"YES" firewall_enable=3D"YES" firewall_nat_enable=3D"YES" ------------- ipfw config, pretty much directly from the handbook: ------------- #!/bin/sh ipfw -q -f flush cmd=3D"ipfw -q add" skip=3D"skipto 1000" pif=3Dgenet0 ks=3D"keep-state" ipfw disable one_pass ipfw -q nat 1 config if $pif same_ports unreg_only reset $cmd 004 allow all from any to any via ue0 # exclude LAN traffic $cmd 005 allow all from any to any via wg0 $cmd 010 allow all from any to any via lo0 # exclude loopback traffic $cmd 099 reass all from any to any in # reassemble inbound packets $cmd 100 nat 1 ip from any to any in via $pif # NAT any inbound packets # Allow the packet through if it has an existing entry in the dynamic rules table $cmd 101 check-state # Authorized outbound packets $cmd 120 $skip udp from any to any out via $pif $ks $cmd 125 $skip tcp from any to any out via $pif setup $ks $cmd 130 $skip icmp from any to any out via $pif $ks # Allow SSH and Wireguard $cmd 200 $skip tcp from any to me 22 setup $ks $cmd 201 $skip udp from any to me 51820 in via $pif $ks $cmd 999 deny log all from any to any $cmd 1000 nat 1 ip from any to any out via $pif $cmd 1001 allow ip from any to any ------------- Speculation: I'm assuming this is the IPv4 version of bug #254060, fixed in https://cgit.freebsd.org/src/commit/sys/arm64/broadcom/genet/if_genet.c?id= =3D2bdcf6237744b2d9d9707d623660d33931daeb52 --=20 You are receiving this mail because: You are the assignee for the bug.=