From nobody Wed May 11 18:47:55 2022 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 640CA1ACD1E1; Wed, 11 May 2022 18:48:09 +0000 (UTC) (envelope-from freebsd@walstatt-de.de) Received: from smtp6.goneo.de (smtp6.goneo.de [85.220.129.31]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Kz3nh142tz4cjB; Wed, 11 May 2022 18:48:07 +0000 (UTC) (envelope-from freebsd@walstatt-de.de) Received: from hub2.goneo.de (hub2.goneo.de [85.220.129.53]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by smtp6.goneo.de (Postfix) with ESMTPS id AEEA410A332B; Wed, 11 May 2022 20:48:00 +0200 (CEST) Received: from hub2.goneo.de (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by hub2.goneo.de (Postfix) with ESMTPS id 130DE10A1E8B; Wed, 11 May 2022 20:47:59 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=walstatt-de.de; s=DKIM001; t=1652294879; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=P/oRS0MTKb2PkQ/83CFeucuXOajJ7ouPhbdJ5iLoSno=; b=HtHzP10bvReq/JbaPmciY7y+9EhdmcxnDwedr0D2LDxUwvWUmc7YaOPdc9o1GxvdLnZC+L OsDBXbcM0KSXSdwPQcVWc+7D6+VEC0E4IwD0WhfpoFsmamDRqv7P3N1quyzqUkL6f8VrmI cWyqi9ejui3t/Yv7e0DCOP0gkrmy57BHxdi66YquooUkc/PqHA/TLVOejSzTlgl0SO7qlY mhLpZ7DbRhsPoJmp9l1Bo87BnfnPdS8TIe4SH1tjV/1bpi7exGtixZIoSILbGIMAtu+Pdq mxnU6mHbbgs18l5I1xU1bskctvOG2e5ul4ZcRr5GhQFisICpAcF+gVsgHR3O0Q== Received: from hermann (dynamic-2a01-0c23-5d0c-3f00-0006-38a3-27c0-2d03.c23.pool.telefonica.de [IPv6:2a01:c23:5d0c:3f00:6:38a3:27c0:2d03]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by hub2.goneo.de (Postfix) with ESMTPSA id 77A2010A1E89; Wed, 11 May 2022 20:47:58 +0200 (CEST) Date: Wed, 11 May 2022 20:47:55 +0200 From: FreeBSD User To: freebsd-jail@freebsd.org, freebsd-net@freebsd.org Subject: Re: FreeBSD 12.3-p5: problems vnet on if_bridge Message-ID: <20220511204755.2028dce9@hermann> In-Reply-To: <20220510212129.35041f02@hermann> References: <20220510212129.35041f02@hermann> List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Rspamd-UID: 7c8d87 X-Rspamd-UID: d44854 X-Rspamd-Queue-Id: 4Kz3nh142tz4cjB X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=walstatt-de.de header.s=DKIM001 header.b=HtHzP10b; dmarc=none; spf=none (mx1.freebsd.org: domain of freebsd@walstatt-de.de has no SPF policy when checking 85.220.129.31) smtp.mailfrom=freebsd@walstatt-de.de X-Spamd-Result: default: False [-1.52 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[walstatt-de.de:s=DKIM001]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; DMARC_NA(0.00)[walstatt-de.de]; NEURAL_SPAM_SHORT(0.38)[0.376]; RCVD_COUNT_THREE(0.00)[4]; DKIM_TRACE(0.00)[walstatt-de.de:+]; RCPT_COUNT_TWO(0.00)[2]; MLMMJ_DEST(0.00)[freebsd-jail,freebsd-net]; R_SPF_NA(0.00)[no SPF record]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; MID_RHS_NOT_FQDN(0.50)[]; ASN(0.00)[asn:25394, ipnet:85.220.128.0/17, country:DE]; RCVD_TLS_ALL(0.00)[]; RCVD_IN_DNSWL_LOW(-0.10)[85.220.129.31:from] X-ThisMailContainsUnwantedMimeParts: N On Tue, 10 May 2022 21:21:29 +0200 FreeBSD User wrote: > Hello, > > I ran into serious trouble setting up a FreeBSD 12.3-RELEASE-p5 host having a second NIC > and vnt jails attached to that second NIC (basically, the host is a recent Xigmanas with > Bastille jails, but the issue also occurs on a vanilla FreeBSD 12.3). > > The host is compromised of two NICs, em0 (management only) and igb0 (service/jails). > Both, the server and the jails as well as the igb0 interface are residing on the same > network, but both NICs are connected to two different ports on a switch, to which we do > not have access (part of the campus infrastructure). > > Both NICs are attached with a IPv4 of the same network, the host is listening on both > NICs for services, i.e. port 22 for ssh. No problem to connect to both(!) addresses via > ssh. igb0 is member of an if_bridge. The box also hosts a bunch of vnet jails, each jail > does have an if_epair created via "jib" and these vnet epairs are members of the bridge, > to which ifb0 is also member. > > Problem: while any service bound to NIC igb0/IPv4 residing on igb0 is accessible > flawlessly, accessing an jail is almost impossible. Pinging a jail does work after a > while the ping initiating host has been waiting, in ery rare situations someone can > access the sshd of the jail, but any access of that kind is highly erratic. From 5 > jails, at most two are responding to pings, the other don't and it is non-deterministic > which host will respond. > > Following some advices found on the web, the following sysctl settings are provided to > if_bridge: > > device if_bridge > net.link.bridge.ipfw: 0 > net.link.bridge.allow_llz_overlap: 0 > net.link.bridge.inherit_mac: 0 > net.link.bridge.log_stp: 0 > net.link.bridge.pfil_local_phys: 0 > net.link.bridge.pfil_member: 0 > net.link.bridge.ipfw_arp: 0 > net.link.bridge.pfil_bridge: 0 > net.link.bridge.pfil_onlyip: 0 > > We do not have access to the switch the box is connected to, so I don't have access to > any logs revealing a problem either to a conceptual misunderstanding of networking of > mine and so a misconfiguration or a probelm with Layer 2 or the switches themselfes. > > I'd like to ask whether someone has a similar setup up and running and could report this > - or give a hint of the problem I possibly made (igb0 is attached to an IPv4 AND is > member of an if_brige on which IPv4 attached vnet jails are residing). > > We have also already setup another "similar" scenarion with the same FreeBSD 12.3-p5 > version and also two NICs, but our "service/jail" NIC is part of a different IPv4 > network and the NIC is attached to a different switch (to which we have full access). > > Thanks in advance, > > O. Hartmann > On FreeBSD 12.3-p5, em0 seems to suffer from a bug regarding hardware chesum support, I see a lot of : [...] Flags [.], cksum 0xe826 (incorrect -> 0x606b), seq 101269476:101270000, ack 5077, win 257, options [nop,nop,TS val 2618589801 ecr 3610923914], length 524 Disabling TXCSUM via "ifconfig em0 -txcsum" renders incorrect -> correct. em0 is: em0@pci0:0:25:0: class=0x020000 card=0x20528086 chip=0x153b8086 rev=0x04 hdr=0x00 vendor = 'Intel Corporation' device = 'Ethernet Connection I217-V' class = network subclass = ethernet bar [10] = type Memory, range 32, base 0xf7d00000, size 131072, enabled bar [14] = type Memory, range 32, base 0xf7d35000, size 4096, enabled bar [18] = type I/O Port, range 32, base 0xf080, size 32, enabled cap 01[c8] = powerspec 2 supports D0 D3 current D0 cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message cap 13[e0] = PCI Advanced Features: FLR TP I remember faintly that there was an issue when I used to use FBSD 12