From nobody Tue May 10 19:21:29 2022 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 288791ACAC0C for ; Tue, 10 May 2022 19:21:48 +0000 (UTC) (envelope-from freebsd@walstatt-de.de) Received: from smtp5.goneo.de (smtp5.goneo.de [85.220.129.30]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4KySZy6RqTz3QPJ for ; Tue, 10 May 2022 19:21:46 +0000 (UTC) (envelope-from freebsd@walstatt-de.de) Received: from hub2.goneo.de (hub2.goneo.de [85.220.129.53]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by smtp5.goneo.de (Postfix) with ESMTPS id E0E1310A0160 for ; Tue, 10 May 2022 21:21:38 +0200 (CEST) Received: from hub2.goneo.de (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by hub2.goneo.de (Postfix) with ESMTPS id 4795210A3317 for ; Tue, 10 May 2022 21:21:37 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=walstatt-de.de; s=DKIM001; t=1652210497; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=mnC4rOg/BfKDawy04gfdbwcl7GDs3Vj2q5sccHBlXxQ=; b=KAlp7tv7+ORfEVUaF9W9kILuiUa35+p4Oav3TosvofJ1Mq1q7IZ0U8q4Orwn5a2Gr7bBq2 owRBlsvJWKB9BsU8dxVyWWJSxhofLuVXaVHFTC8S0NNyAz1TroHKAEp9Mh6MCUmhf+zKL+ qABIjTU2vf+GaZQELnf4kVLPWmCGhHtXwBEZiVCJk+V6GNirHztHQBVBWhfSECvr1tQLDr b6kVJrRY44iJJPAzMvI1Cucb0EKP+Bwxb3y1eSd7iRYwY5p/A5o+EAicAAZUQ9hfXLIkQv zaOD+c9A4dybjzOQhxG2hUBONMjru6/2PPbFStxlzricNQGQz5BG1itCE8E59g== Received: from hermann (dynamic-2a01-0c22-ad27-3c00-f407-9de1-b468-ac89.c22.pool.telefonica.de [IPv6:2a01:c22:ad27:3c00:f407:9de1:b468:ac89]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by hub2.goneo.de (Postfix) with ESMTPSA id 170AF10A3308 for ; Tue, 10 May 2022 21:21:37 +0200 (CEST) Date: Tue, 10 May 2022 21:21:29 +0200 From: FreeBSD User To: freebsd-jail@freebsd.org Subject: FreeBSD 12.3-p5: problems vnet on if_bridge Message-ID: <20220510212129.35041f02@hermann> List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Rspamd-UID: a0f323 X-Rspamd-UID: fceb88 X-Rspamd-Queue-Id: 4KySZy6RqTz3QPJ X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=walstatt-de.de header.s=DKIM001 header.b=KAlp7tv7; dmarc=none; spf=none (mx1.freebsd.org: domain of freebsd@walstatt-de.de has no SPF policy when checking 85.220.129.30) smtp.mailfrom=freebsd@walstatt-de.de X-Spamd-Result: default: False [-2.89 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[walstatt-de.de:s=DKIM001]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-jail@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; RCVD_COUNT_THREE(0.00)[4]; DMARC_NA(0.00)[walstatt-de.de]; DKIM_TRACE(0.00)[walstatt-de.de:+]; NEURAL_HAM_SHORT(-0.99)[-0.987]; MLMMJ_DEST(0.00)[freebsd-jail]; R_SPF_NA(0.00)[no SPF record]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; MID_RHS_NOT_FQDN(0.50)[]; ASN(0.00)[asn:25394, ipnet:85.220.128.0/17, country:DE]; RCVD_TLS_ALL(0.00)[]; RCVD_IN_DNSWL_LOW(-0.10)[85.220.129.30:from] X-ThisMailContainsUnwantedMimeParts: N Hello, I ran into serious trouble setting up a FreeBSD 12.3-RELEASE-p5 host having a second NIC and vnt jails attached to that second NIC (basically, the host is a recent Xigmanas with Bastille jails, but the issue also occurs on a vanilla FreeBSD 12.3). The host is compromised of two NICs, em0 (management only) and igb0 (service/jails). Both, the server and the jails as well as the igb0 interface are residing on the same network, but both NICs are connected to two different ports on a switch, to which we do not have access (part of the campus infrastructure). Both NICs are attached with a IPv4 of the same network, the host is listening on both NICs for services, i.e. port 22 for ssh. No problem to connect to both(!) addresses via ssh. igb0 is member of an if_bridge. The box also hosts a bunch of vnet jails, each jail does have an if_epair created via "jib" and these vnet epairs are members of the bridge, to which ifb0 is also member. Problem: while any service bound to NIC igb0/IPv4 residing on igb0 is accessible flawlessly, accessing an jail is almost impossible. Pinging a jail does work after a while the ping initiating host has been waiting, in ery rare situations someone can access the sshd of the jail, but any access of that kind is highly erratic. From 5 jails, at most two are responding to pings, the other don't and it is non-deterministic which host will respond. Following some advices found on the web, the following sysctl settings are provided to if_bridge: device if_bridge net.link.bridge.ipfw: 0 net.link.bridge.allow_llz_overlap: 0 net.link.bridge.inherit_mac: 0 net.link.bridge.log_stp: 0 net.link.bridge.pfil_local_phys: 0 net.link.bridge.pfil_member: 0 net.link.bridge.ipfw_arp: 0 net.link.bridge.pfil_bridge: 0 net.link.bridge.pfil_onlyip: 0 We do not have access to the switch the box is connected to, so I don't have access to any logs revealing a problem either to a conceptual misunderstanding of networking of mine and so a misconfiguration or a probelm with Layer 2 or the switches themselfes. I'd like to ask whether someone has a similar setup up and running and could report this - or give a hint of the problem I possibly made (igb0 is attached to an IPv4 AND is member of an if_brige on which IPv4 attached vnet jails are residing). We have also already setup another "similar" scenarion with the same FreeBSD 12.3-p5 version and also two NICs, but our "service/jail" NIC is part of a different IPv4 network and the NIC is attached to a different switch (to which we have full access). Thanks in advance, O. Hartmann