From nobody Sun May 15 11:40:57 2022 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id DD5FC1AE6604 for ; Sun, 15 May 2022 11:41:02 +0000 (UTC) (envelope-from freebsd@walstatt-de.de) Received: from smtp6.goneo.de (smtp6.goneo.de [85.220.129.31]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4L1L716SNMz3vkL for ; Sun, 15 May 2022 11:41:01 +0000 (UTC) (envelope-from freebsd@walstatt-de.de) Received: from hub1.goneo.de (hub1.goneo.de [85.220.129.52]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by smtp6.goneo.de (Postfix) with ESMTPS id B570510A332B for ; Sun, 15 May 2022 13:40:59 +0200 (CEST) Received: from hub1.goneo.de (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by hub1.goneo.de (Postfix) with ESMTPS id 1846B10A32F4 for ; Sun, 15 May 2022 13:40:58 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=walstatt-de.de; s=DKIM001; t=1652614858; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=41MfO8vesY3I6akHKnEvpWmkrDeFG54RrxlA8Ra9404=; b=iE+ujIefY4Eba8RI03JiqMJoUw9+ZmpbbdKzWu8ZJ+rAyDOJeh9x9Bp0geUsJ/1fBzdNZl vL7RCZ0lNEeaOIWTSp7s8IGOHY9+AFw154FAxzICw6peSjyYflkr17OkYvVIlcPcdnJAFQ ufuIV/AHRmMwvgIvsSvepBhWtvo01wCWeJHMf7PyEb0puTi5ZsDx2W1QMfPl5sH8LfNPCF tj5J/ybZv9jdowtDuT29SeD4MKN+m8awv9RG6/IXS1omoTYFtcjAfmEzblQOLAuMqKiqOt vGHH35JDF2xjViBHyyatw0UrJ9lUOSMU4j3qSd1MvVL7I3pBOimRiob/OUQb+Q== Received: from hermann (dynamic-2a01-0c23-64b5-f700-711a-9937-bc6c-d4d8.c23.pool.telefonica.de [IPv6:2a01:c23:64b5:f700:711a:9937:bc6c:d4d8]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by hub1.goneo.de (Postfix) with ESMTPSA id E1A3410A330F for ; Sun, 15 May 2022 13:40:57 +0200 (CEST) Date: Sun, 15 May 2022 13:40:57 +0200 From: FreeBSD User To: freebsd-net@freebsd.org Subject: FreeBSD serious problems with vnet on if_bridge (probably ARP related?) Message-ID: <20220515134057.6906cf76@hermann> List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Rspamd-UID: 9ed4a2 X-Rspamd-UID: b24b55 X-Rspamd-Queue-Id: 4L1L716SNMz3vkL X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=walstatt-de.de header.s=DKIM001 header.b=iE+ujIef; dmarc=none; spf=none (mx1.freebsd.org: domain of freebsd@walstatt-de.de has no SPF policy when checking 85.220.129.31) smtp.mailfrom=freebsd@walstatt-de.de X-Spamd-Result: default: False [-2.87 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[walstatt-de.de:s=DKIM001]; NEURAL_HAM_MEDIUM(-0.97)[-0.970]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; RCVD_COUNT_THREE(0.00)[4]; DMARC_NA(0.00)[walstatt-de.de]; DKIM_TRACE(0.00)[walstatt-de.de:+]; NEURAL_HAM_SHORT(-1.00)[-1.000]; MLMMJ_DEST(0.00)[freebsd-net]; R_SPF_NA(0.00)[no SPF record]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; MID_RHS_NOT_FQDN(0.50)[]; ASN(0.00)[asn:25394, ipnet:85.220.128.0/17, country:DE]; RCVD_TLS_ALL(0.00)[]; SUBJECT_HAS_QUESTION(0.00)[]; RCVD_IN_DNSWL_LOW(-0.10)[85.220.129.31:from] X-ThisMailContainsUnwantedMimeParts: N Hello, I ran into serious trouble setting up a FreeBSD 12.3-RELEASE-p5 host having a second NIC and vnet jails attached to that second NIC (basically, the host is a recent Xigmanas with Bastille jails). Hopefully, I can gather some answeres here. The host is comprised of two NICs, em0 (management only) and igb0 (service/jails). Both, the server and the jails as well as the igb0 interface are residing on the same network, but both NICs are connected to two different ports on the switch (I assume, we do not have access to the campus network inrastructure). Both NICs are attached with an IPv4 of the same network, the host is listening on both NICs for services, port 22 (ssh) for instance, other services are supposed to be bound to the second NIC (igb0, like NFS, SMB and so on). em0 is only listening for 22/tcp and 443/tcp as it is meant for management only. igb0 is member of a bridge as well as the vnet interfaces (epair) of thr jails, created via "jib" (the host is basically a recent XigmaNAS 12.3.0.4.9047). The igb0 NIC has also an IP from the LAN - some advices to do so, other advice to avoid setting an IP, but the management interface of XigmaNAS forces me to apply an IP. Just for the record. Problem: it seems that in a non predictable way connections are droped, ARP packages reach the vnet and in other cases ARP doesn't reach a vnet. The phenomenon is weird. The host is running 6 jails with regular FQDN and IPv4 addresses. All jails are up, the base NIC (igb0) is also up and running. It is possible, in rare cases, to connect via ssh frome remote sites to at most two(!) of the jails. It is impossible to predict which jail is connecting first and it is like a lottery which one will respond. Pinging the jail from the LAN or remote sites is also weird: as with the ssh connection, sometimes a jail responds immediately as expected, in some other cases it takes 10 - 30 seconds until ping starts to report replies. Connecting to the base host has never been a problem, so i assume the base network to be all right. Connecting to a jail locally via "jexec -l" and performing some tcpdump invstigations reveals weird results: tcpdump -vi vnet0 arp does show ARP a plethora of packets from the local network on a jail that is reposnding quickly to ping or is giving access to ssh, but on those jails which are resilient to connection and ping attempts, ARP packets seem to vanish. By having screens/terminals adjacent of those jails while pinnging and tcpdumping, its obvious that not all "arp: who-has ... tell" requests are evenly distributed to all bridge members as I'd expect. Following some advices found on the web, the following sysctl settings are provided to if_bridge (main host): device if_bridge net.link.bridge.ipfw: 0 net.link.bridge.allow_llz_overlap: 0 net.link.bridge.inherit_mac: 0 net.link.bridge.log_stp: 0 net.link.bridge.pfil_local_phys: 0 net.link.bridge.pfil_member: 0 net.link.bridge.ipfw_arp: 0 net.link.bridge.pfil_bridge: 0 net.link.bridge.pfil_onlyip: 0 I also fiddled around with disabling LRO, TSO, RXCSUM and TXCSUM on all physicsal NICs as somewhere recommended due to buggy FreeBSD NIC driver, but without any help. Please see below for some physical details to the NICs on the box. In another department I've setup a similar box (also XigmaNAS), but this time the second NIC is a different type (i350-T2). The physical port which is member of the if_bridge on which the vnet epair of the jails are residing, is also setup with an IPv4 address (as described above), but member of a different network and does not share the same switch/network with the dedicated management port. This box's jails are acting and responding as expected. Thanks in advance, O. Hartmann igb0@pci0:4:0:0: class=0x020000 card=0x00028086 chip=0x15338086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = 'I210 Gigabit Network Connection' class = network subclass = ethernet bar [10] = type Memory, range 32, base 0xf7900000, size 1048576, enabled bar [1c] = type Memory, range 32, base 0xf7a00000, size 16384, enabled cap 01[40] = powerspec 3 supports D0 D3 current D0 cap 05[50] = MSI supports 1 message, 64 bit, vector masks cap 11[70] = MSI-X supports 5 messages, enabled Table in map 0x1c[0x0], PBA in map 0x1c[0x2000] cap 10[a0] = PCI-Express 2 endpoint max data 128(512) FLR NS max read 512 link x1(x1) speed 2.5(2.5) ASPM L1(L0s/L1) ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected ecap 0003[140] = Serial 1 somenumber ecap 0017[1a0] = TPH Requester 1 m0@pci0:0:25:0: class=0x020000 card=0x20528086 chip=0x153b8086 rev=0x04 hdr=0x00 vendor = 'Intel Corporation' device = 'Ethernet Connection I217-V' class = network subclass = ethernet bar [10] = type Memory, range 32, base 0xf7d00000, size 131072, enabled bar [14] = type Memory, range 32, base 0xf7d35000, size 4096, enabled bar [18] = type I/O Port, range 32, base 0xf080, size 32, enabled cap 01[c8] = powerspec 2 supports D0 D3 current D0 cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message cap 13[e0] = PCI Advanced Features: FLR TP