From nobody Sun Aug 28 16:56:43 2022 X-Original-To: freebsd-ipfw@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4MG08s6VK8z4ZxtQ for ; Sun, 28 Aug 2022 16:56:45 +0000 (UTC) (envelope-from 01000182e5625776-1db6fdaa-39b3-42ac-9320-a1ed901e381e-000000@amazonses.com) Received: from a8-200.smtp-out.amazonses.com (a8-200.smtp-out.amazonses.com [54.240.8.200]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4MG08r4hk7z3N3g for ; Sun, 28 Aug 2022 16:56:44 +0000 (UTC) (envelope-from 01000182e5625776-1db6fdaa-39b3-42ac-9320-a1ed901e381e-000000@amazonses.com) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=6gbrjpgwjskckoa6a5zn6fwqkn67xbtw; d=amazonses.com; t=1661705803; h=From:To:Subject:Date:Message-ID:MIME-Version:Content-Transfer-Encoding:Content-Type:Feedback-ID; bh=GWQDh9ivwofkpF0Lw5FTYy4Gcg6MWhf2UfaPt8NsyzQ=; b=HSkAcg3pomsVaolcJae9eUdRK6xoElXwnBdFCNEWTEaYdhQGCc/N9Mqp+n/fKBzs KnRAcGtLbj8ubf4ERc95ocHj+kaGaP2q85y3xGcyU0LWv797qv3UlN0LRJtDbn5QtLC slwMPrNjU5aW6QiPMjpOIoy2xZvCQBitoItlHCPM= From: Steven To: freebsd-ipfw@freebsd.org Subject: ipfw fwd not forwarding correctly Date: Sun, 28 Aug 2022 16:56:43 +0000 Message-ID: <01000182e5625776-1db6fdaa-39b3-42ac-9320-a1ed901e381e-000000@email.amazonses.com> List-Id: IPFW Technical Discussions List-Archive: https://lists.freebsd.org/archives/freebsd-ipfw List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-ipfw@freebsd.org MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" Feedback-ID: 1.us-east-1.6/5hyFEoYVzl3Z8NRna5TO5n3ZbRKlo4lqs9bwHt4SA=:AmazonSES X-SES-Outgoing: 2022.08.28-54.240.8.200 X-Rspamd-Queue-Id: 4MG08r4hk7z3N3g X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=amazonses.com header.s=6gbrjpgwjskckoa6a5zn6fwqkn67xbtw header.b=HSkAcg3p; dmarc=none; spf=pass (mx1.freebsd.org: domain of 01000182e5625776-1db6fdaa-39b3-42ac-9320-a1ed901e381e-000000@amazonses.com designates 54.240.8.200 as permitted sender) smtp.mailfrom=01000182e5625776-1db6fdaa-39b3-42ac-9320-a1ed901e381e-000000@amazonses.com X-Spamd-Result: default: False [-3.20 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; NEURAL_HAM_SHORT(-1.00)[-0.998]; FORGED_SENDER(0.30)[steven@bsd-ipfw.sdwmail.org,01000182e5625776-1db6fdaa-39b3-42ac-9320-a1ed901e381e-000000@amazonses.com]; R_DKIM_ALLOW(-0.20)[amazonses.com:s=6gbrjpgwjskckoa6a5zn6fwqkn67xbtw]; R_SPF_ALLOW(-0.20)[+ip4:54.240.0.0/18:c]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[freebsd-ipfw@freebsd.org]; RCVD_COUNT_ZERO(0.00)[0]; RWL_MAILSPIKE_POSSIBLE(0.00)[54.240.8.200:from]; MIME_TRACE(0.00)[0:+]; RCVD_IN_DNSWL_NONE(0.00)[54.240.8.200:from]; ASN(0.00)[asn:14618, ipnet:54.240.8.0/21, country:US]; ARC_NA(0.00)[]; FROM_NEQ_ENVFROM(0.00)[steven@bsd-ipfw.sdwmail.org,01000182e5625776-1db6fdaa-39b3-42ac-9320-a1ed901e381e-000000@amazonses.com]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; DKIM_TRACE(0.00)[amazonses.com:+]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; DMARC_NA(0.00)[sdwmail.org]; DWL_DNSWL_NONE(0.00)[amazonses.com:dkim] X-ThisMailContainsUnwantedMimeParts: N SUMMARY: Incoming connections do not forward initial [SYN, ACK] packet o= n=20 incoming connections to the tunnel device. The very final [ACK] after a t= cp=20 session close is also not forwarded to the tunnel device. This is accordi= ng=20 to the ipfw fwd rule. Instead, certain packet gets forwarded to the defaul= t=20 route out my ethernet nic, re0, even with a matching ipfw fwd rule hit. This has an effect of not allowing incoming connections and incorrect tcp= =20 transmission behavior on outgoing connections. using: =46reeBSD steven13.1-RELEASE-p1 FreeBSD 13.1-RELEASE-p1 GENERIC amd64 I have set up a VPN tunnel using interface tun1. The test tcp seesion is (a) an outgoing ssh session (binding to the local t= un1=20 IP) as well as (b) an incoming ssh session back to the tun1 IP port 22. tun1: flags=3D8051 metric 0 mtu 1500 options=3D80000 inet 192.121.246.12 --> 192.121.246.1 netmask 0xffffffc0 groups: tun nd6 options=3D29 Internet: Destination Gateway Flags Netif Expire default 10.2.2.1 UGS re0 10.2.2.0/23 link#1 U re0 10.2.2.2 link#1 UHS lo0 192.121.246.0/26 192.121.246.1 UGS tun1 192.121.246.1 link#5 UH tun1 192.121.246.12 link#5 UHS lo0 ipfw rules: 48000 6 402 fwd 192.121.246.1 ip from 192.121.246.12 22 to any o= ut=20 xmit tun1 48500 30 1800 fwd 192.121.246.1 ip from 192.121.246.12 22 to any o= ut=20 xmit re0 49000 133 7980 fwd 192.121.246.1 ip from 192.121.246.12 22 to any o= ut 50000 6921 4255710 fwd 192.121.246.1 ip from 192.121.246.0/26 to any out [Initially I started with rule 50000 but made more specific rules to see wh= ere=20 it gets "hit"] Now, if I add the "VPN bypass" routes: route add -net 0.0.0.0/1 192.121.246.1 route add -net 128.0.0.0/1 192.121.246.1 when I telnet back to my hose from the outside: telnet 192.121.246.12 22 All will work fine. In this case, rule 48000 will be hit Otherwise, the default route will be used and sent through 10.2.2.2, re0 an= d=20 ipfw rule 48500 will be hit. =46urthermore, without the "VPN bypass" routes and just the default route, a normal outgoing tcp session will work fine until the very end. Initia= lly=20 all packets will be sent and received through the tunnel interface and hit = on=20 the ipfw fwd rule. However, the very last packet, on tcp close, will be se= nt=20 through the default route, re0 instead. sent through tun1: 49 18.123367 192.121.246.12 128.200.85.17 TCP 56 =20 24761 =E2=86=92 22 [FIN, ACK] Seq=3D4583 Ack=3D4950 Win=3D66496 Len=3D0 TSv= al=3D3821441136=20 TSecr=3D2539456840 50 18.216585 128.200.85.17 192.121.246.12 TCP 56 =20 22 =E2=86=92 24761 [ACK] Seq=3D4950 Ack=3D4583 Win=3D33408 Len=3D0 TSval=3D= 2539456938=20 TSecr=3D3821441136 51 18.218972 128.200.85.17 192.121.246.12 TCP 56 =20 22 =E2=86=92 24761 [FIN, ACK] Seq=3D4950 Ack=3D4584 Win=3D33408 Len=3D0 TSv= al=3D2539456940=20 TSecr=3D3821441136 last [ACK] packet, after the above sequence, sent through re0: 1 0.000000 192.121.246.12 128.200.85.17 TCP 70 =20 53986 =E2=86=92 22 [ACK] Seq=3D1 Ack=3D1 Win=3D1038 Len=3D0 TSval=3D3860450= 619 TSecr=3D2539249268 The above seems to be proof that something is awry. Why would all the=20 incoming and outgoing packets, with the same source IP,source port and dest= =20 IP,dest port go through the proper interface, tun1, but then the very last= =20 [ACK] packet go someplace else? The manpage states, If ipaddr is not a local address, then the port number (if specified) is ignored, and the packet will be forwarded to the remote address, using the route as found in the local routing table for that IP. The ipfw fwd rule is being hit, it suppose to be forwarded to 192.121.246.1= ,=20 the "192.121.246.1 link#5 UH tun1" route entry=20 should apply and be sent to tun1 instead of re0 default. I just started using ipfw, so maybe I have overlooked something. But afte= r=20 capturing packets and seeing the irregular behaviour I think something may = be=20 wrong. Please let me know if I have overlooked something. Steven