From nobody Tue May 24 10:00:37 2022 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id DF77C1AE8F29; Tue, 24 May 2022 10:00:51 +0000 (UTC) (envelope-from ol@dbconn.net) Received: from mout-b-110.mailbox.org (mout-b-110.mailbox.org [IPv6:2001:67c:2050:102:465::110]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4L6qTH0hgqz3KNc; Tue, 24 May 2022 10:00:51 +0000 (UTC) (envelope-from ol@dbconn.net) Received: from smtp202.mailbox.org (smtp202.mailbox.org [IPv6:2001:67c:2050:b231:465::202]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mout-b-110.mailbox.org (Postfix) with ESMTPS id 4L6qT64Zndz9sRB; Tue, 24 May 2022 12:00:42 +0200 (CEST) Date: Tue, 24 May 2022 12:00:37 +0200 From: Ole Lemke To: FreeBSD User Cc: freebsd-jail@freebsd.org, freebsd-net@freebsd.org Subject: Re: FreeBSD 12.3-p5: problems vnet on if_bridge Message-ID: <20220524120037.46b49baa@lenp43s> In-Reply-To: <20220511204755.2028dce9@hermann> References: <20220510212129.35041f02@hermann> <20220511204755.2028dce9@hermann> List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 Content-Type: multipart/signed; boundary="Sig_//J.G76yaS6QF16AnR+droUK"; protocol="application/pgp-signature"; micalg=pgp-sha512 X-Rspamd-Queue-Id: 4L6qTH0hgqz3KNc X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of ol@dbconn.net designates 2001:67c:2050:102:465::110 as permitted sender) smtp.mailfrom=ol@dbconn.net X-Spamd-Result: default: False [-4.89 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.997]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2001:67c:2050::/48:c]; NEURAL_HAM_LONG(-0.99)[-0.993]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; MID_RHS_NOT_FQDN(0.50)[]; DMARC_NA(0.00)[dbconn.net]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-1.00)[-1.000]; MLMMJ_DEST(0.00)[freebsd-jail,freebsd-net]; SIGNED_PGP(-2.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:199118, ipnet:2001:67c:2050::/48, country:DE]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] X-ThisMailContainsUnwantedMimeParts: N --Sig_//J.G76yaS6QF16AnR+droUK Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable Hello, could you solve the problem? I think I ran into the same problem. I opened a Ticket. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D264198 I seems to be related to IPFW and effects vnet-Jails and also bhyve VMs. regards Ole Wed, 11 May 2022 20:47:55 +0200 - FreeBSD User : > On Tue, 10 May 2022 21:21:29 +0200 > FreeBSD User wrote: >=20 > > Hello, > >=20 > > I ran into serious trouble setting up a FreeBSD 12.3-RELEASE-p5 > > host having a second NIC and vnt jails attached to that second NIC > > (basically, the host is a recent Xigmanas with Bastille jails, but > > the issue also occurs on a vanilla FreeBSD 12.3). > >=20 > > The host is compromised of two NICs, em0 (management only) and igb0 > > (service/jails). Both, the server and the jails as well as the igb0 > > interface are residing on the same network, but both NICs are > > connected to two different ports on a switch, to which we do not > > have access (part of the campus infrastructure). > >=20 > > Both NICs are attached with a IPv4 of the same network, the host is > > listening on both NICs for services, i.e. port 22 for ssh. No > > problem to connect to both(!) addresses via ssh. igb0 is member of > > an if_bridge. The box also hosts a bunch of vnet jails, each jail > > does have an if_epair created via "jib" and these vnet epairs are > > members of the bridge, to which ifb0 is also member. > >=20 > > Problem: while any service bound to NIC igb0/IPv4 residing on igb0 > > is accessible flawlessly, accessing an jail is almost impossible. > > Pinging a jail does work after a while the ping initiating host has > > been waiting, in ery rare situations someone can access the sshd of > > the jail, but any access of that kind is highly erratic. From 5 > > jails, at most two are responding to pings, the other don't and it > > is non-deterministic which host will respond.=20 > >=20 > > Following some advices found on the web, the following sysctl > > settings are provided to if_bridge:=20 > >=20 > > device if_bridge > > net.link.bridge.ipfw: 0 > > net.link.bridge.allow_llz_overlap: 0 > > net.link.bridge.inherit_mac: 0 > > net.link.bridge.log_stp: 0 > > net.link.bridge.pfil_local_phys: 0 > > net.link.bridge.pfil_member: 0 > > net.link.bridge.ipfw_arp: 0 > > net.link.bridge.pfil_bridge: 0 > > net.link.bridge.pfil_onlyip: 0 > >=20 > > We do not have access to the switch the box is connected to, so I > > don't have access to any logs revealing a problem either to a > > conceptual misunderstanding of networking of mine and so a > > misconfiguration or a probelm with Layer 2 or the switches > > themselfes. > >=20 > > I'd like to ask whether someone has a similar setup up and running > > and could report this > > - or give a hint of the problem I possibly made (igb0 is attached > > to an IPv4 AND is member of an if_brige on which IPv4 attached vnet > > jails are residing). > >=20 > > We have also already setup another "similar" scenarion with the > > same FreeBSD 12.3-p5 version and also two NICs, but our > > "service/jail" NIC is part of a different IPv4 network and the NIC > > is attached to a different switch (to which we have full access). > >=20 > > Thanks in advance, > >=20 > > O. Hartmann > >=20 >=20 > On FreeBSD 12.3-p5, em0 seems to suffer from a bug regarding hardware > chesum support, I see a lot of : > [...] > Flags [.], cksum 0xe826 (incorrect -> 0x606b), seq > 101269476:101270000, ack 5077, win 257, options [nop,nop,TS val > 2618589801 ecr 3610923914], length 524 >=20 > Disabling TXCSUM via "ifconfig em0 -txcsum" renders incorrect -> > correct. >=20 > em0 is: >=20 > em0@pci0:0:25:0: class=3D0x020000 card=3D0x20528086 > chip=3D0x153b8086 rev=3D0x04 hdr=3D0x00 vendor =3D 'Intel Corporation' > device =3D 'Ethernet Connection I217-V' > class =3D network > subclass =3D ethernet > bar [10] =3D type Memory, range 32, base 0xf7d00000, size 131072, > enabled bar [14] =3D type Memory, range 32, base 0xf7d35000, size > 4096, enabled bar [18] =3D type I/O Port, range 32, base 0xf080, size > 32, enabled cap 01[c8] =3D powerspec 2 supports D0 D3 current D0 > cap 05[d0] =3D MSI supports 1 message, 64 bit enabled with 1 message > cap 13[e0] =3D PCI Advanced Features: FLR TP >=20 >=20 > I remember faintly that there was an issue when I used to use FBSD 12 >=20 >=20 >=20 >=20 --Sig_//J.G76yaS6QF16AnR+droUK Content-Type: application/pgp-signature Content-Description: Digitale Signatur von OpenPGP -----BEGIN PGP SIGNATURE----- iQIzBAEBCgAdFiEEcfiBcfJXFWtoGXjffPhD3vCz2EwFAmKMrMUACgkQfPhD3vCz 2EzuvQ//R896neYnAL0V0sAXHkXspYDulmKBD1vS4POImoc10ATP9nOfE55jIiOr Uvdwq9cImshcH8FHC3Ozf8QYfrewmXHMX6eZTjSt99qwJggDVDXJXtLOQxMZbL7w 5+suO4gxbmFCcXjeqHen8oxKNgu9wcoFxC2e2p2E/z41oN69iviF8tV+WE9qwWlI 4OsyN0KpttgA5L12dk81pOMnu46nkpZ7fH0QSW77Q/qR58bpjMFCPx5OBvMySdN/ 25en208W/TAOwvTl1W7qFDX6RmEUbSp16CrijlgeRU8kSny9ttIe/sMzxDMo3dCZ 9FD8DpsBkRQnQYKg98aNmC/gzzklQvkoAXi+O/KRUrNkCzJGObfvF6cykusjLYgH vYD7yjUpMqtiSY7OmyWWrNJJvPLX9zQ3qhhWinZw9RetGhe6eWfGJkLJYwRkBaIv eNjaqL+2ru/o7aHZNlioMfLPko5xCoAQr4NfT4eF0GXR433851ngxbsNMsRbzx7g TDEpOYtlUxX/jJEQeurjQc5Ymay8/rFDMIeOM9nuBlBeZkmxxy/wqhhF0qUHPYfH L5Y8SZeaw9QZHZ7zsuMT4bJj/TbokyeHnQ2S4aQjCFCCq56CKFhgyHV4WSxJvwsV 7Ft6UAUOQnhvBB+K7DqHHXXiRWBKo7afkGWtwoHYDf4XJbkpvUo= =EyM7 -----END PGP SIGNATURE----- --Sig_//J.G76yaS6QF16AnR+droUK--