From nobody Fri Mar 04 14:36:16 2022 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 5249519F86A7 for ; Fri, 4 Mar 2022 14:36:24 +0000 (UTC) (envelope-from joh.hendriks@gmail.com) Received: from mail-ej1-x631.google.com (mail-ej1-x631.google.com [IPv6:2a00:1450:4864:20::631]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4K99Qb6Gnhz3MDB for ; Fri, 4 Mar 2022 14:36:23 +0000 (UTC) (envelope-from joh.hendriks@gmail.com) Received: by mail-ej1-x631.google.com with SMTP id r13so17932589ejd.5 for ; Fri, 04 Mar 2022 06:36:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:date:mime-version:user-agent:content-language:to:from :subject:content-transfer-encoding; bh=/7KpEosJGc/+jIgcVgJetmkT38TRZr7SOhYCf+WJJjY=; b=Rc5IxfRe+1Xm/SI8X4U6yYmPJi5pQEZLoBHb8jTHWmZflYLvedghYMYbnbZtI7FufH Oxt654K3kKHXY3Nna0XU6I91jpPOwkPvwcUWUQmE0tDp3os+E/jL7oSzdzJyVVXbB5oQ VeYFO+LkmnRGIByjhsJSbzLeAsejek/YeuHyGr6TfknUFxfxb3crG8YODvoN2Y7UXwlm 82ri/Z8ciBH6eyeIdqfIBZYBRJalDms3InoCWLk37XmAI962eq8k1CHQmol85RlaxDMd 7N9EzfTz/NMGT7VOHHp+fDq2uUYMdnQ0qv6FJAaHaBcSLQtn84VrtTyHybpoInFIGKOz EVcg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:date:mime-version:user-agent :content-language:to:from:subject:content-transfer-encoding; bh=/7KpEosJGc/+jIgcVgJetmkT38TRZr7SOhYCf+WJJjY=; b=SauHeVwhTX2aihKWTmeSSMhDMOljeAeveX0N+o9C3b6nHzVFnjeXEEBt+7Tgmq1d7j Kfd9vdFQvCdYibIEpTPsGLXcPlACdOv0ywkSdSy7xbpv/hs2HyzjnBMcAtlMrE7zFd3Q iv9tiBXf6YQ9bXL7hbJv3CgfwGb418ebcBikc22nFASBivhSkfO26J6tP1KGkAeHQ3mu YCwmTb0inUQMp4C0Z0Uvx6KeYXCse+PuVd3qrnMVyH6TuJNZ/eynNjcsk7vbKCEOsXWG pkzbb9TchK7rQz2EGQYZUYF98o+a302nL21Uj466G3N6ZzZe+TOWRx7iEwp2+1gnjhr8 WUsQ== X-Gm-Message-State: AOAM531XGWxEqHKlSoy7l2bHdXpQfih84WwF3iSIEVVwQ1GUUg1TP+Sx CQYJ+Fw2CiM2aoaVU506weYVyvrOIxI= X-Google-Smtp-Source: ABdhPJywLs5KUs7wtG0leGgoaXUYRuub2Vzw0vUM2NsgmDzSmqAE4IepI5yTEOF2T0I//FbUAkAyZw== X-Received: by 2002:a17:906:16cc:b0:6ce:e607:ff02 with SMTP id t12-20020a17090616cc00b006cee607ff02mr30315975ejd.418.1646404582481; Fri, 04 Mar 2022 06:36:22 -0800 (PST) Received: from [192.168.1.24] (85-147-130-226.cable.dynamic.v4.ziggo.nl. [85.147.130.226]) by smtp.gmail.com with ESMTPSA id e12-20020a056402190c00b0041615cd434csm62271edz.60.2022.03.04.06.36.21 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 04 Mar 2022 06:36:21 -0800 (PST) Message-ID: Date: Fri, 4 Mar 2022 15:36:16 +0100 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Thunderbird/91.6.1 Content-Language: en-US To: FreeBSD Current From: Johan Hendriks Subject: vnet jails loose network connectivity Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 4K99Qb6Gnhz3MDB X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=Rc5IxfRe; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of johhendriks@gmail.com designates 2a00:1450:4864:20::631 as permitted sender) smtp.mailfrom=johhendriks@gmail.com X-Spamd-Result: default: False [-3.93 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.93)[-0.926]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; TAGGED_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::631:from]; MLMMJ_DEST(0.00)[freebsd-current]; RCVD_TLS_ALL(0.00)[] X-ThisMailContainsUnwantedMimeParts: N Hello all, i use jails for some testing, but i can not seem to make it stable. I use vnet jails with a bridge but when i put some load on it, some jails loose there network connectivity. My setup is as follows, haproxy internal IP 10.233.185.20 using binat to make it Public accessable. Then a varnish jail, and two web servers al on the 10.233.185.x range. If i give it a little load with hey (hey -h2 -n 10 -c 20 -z 60s https://wp.test.nl) than within the test the haproxy jail is not reachable anymore it is not pingable from the host machine, and from the other jails. restarting the jails solves it, if i leave the system alone for some time i saw the varnish jail become unresponsive. If i do a tcpdump on the epair${name}a interface i do see the packages from the host machine to the jail but the jail itself is not reachable. There is nothing in the logs from the host and the jail itself, i can ping the jails ip adres from the jail itself. I do not think i have a special setup, but i could be doing something wrong. my jail.conf # Global settings applied to all jails. $domain = "test.nl"; $subdomain = ""; exec.start = "/bin/sh /etc/rc"; exec.stop = "/bin/sh /etc/rc.shutdown"; exec.clean; mount.fstab = "/storage/jails/$name.fstab"; exec.system_user  = "root"; exec.jail_user    = "root"; mount.devfs; sysvshm="new"; sysvsem="new"; allow.raw_sockets; allow.set_hostname = 0; allow.sysvipc; enforce_statfs = "2"; devfs_ruleset     = "11"; path = "/storage/jails/${name}"; host.hostname = "${name}${subdomain}.${domain}"; # Networking $uplinkdev        = "vtnet1"; $epid             = "${ip}"; $subnet           = "10.233.185."; $cidr             = "/24"; $ipv4_addr        = "${subnet}${ip}${cidr}"; vnet; vnet.interface    = "vnet0"; $epair=epair${ip}; vnet; #vnet.interface    = "${epair}b";  # default vnet interface exec.prestart     = "ifconfig bridge0 > /dev/null 2>&1 || ( ifconfig bridge0 create up && ifconfig bridge0 addm $uplinkdev )"; exec.prestart    += "ifconfig ${epair} create up description jail_${name}   || echo 'Skipped creating epair (exists?)'"; exec.prestart    += "ifconfig bridge0 addm ${epair}a           || echo 'Skipped adding bridge member (already member?)'"; exec.created      = "ifconfig ${epair}b name vnet0"; exec.start        = "/bin/sh /etc/rc"; exec.consolelog   = "/var/log/jail/$name.test.nl"; exec.stop         = "/bin/sh /etc/rc.shutdown"; exec.poststop     = "ifconfig bridge0 deletem ${epair}a"; exec.poststop    += "ifconfig ${epair}a destroy"; varnish01 {     $ip = 16;     mount.fstab = "";     path = "/storage/jails/${name}"; } web01 {     $ip = 18; } web02 {     $ip = 19; } haproxy {     $ip = 20;     mount.fstab = "";     path = "/storage/jails/${name}"; } My ifconfig bridge0: flags=8843 metric 0 mtu 1500     ether 58:9c:fc:10:ff:82     inet 10.233.185.1 netmask 0xffffff00 broadcast 10.233.185.255     id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15     maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200     root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0     member: epair20a flags=143             ifmaxaddr 0 port 13 priority 128 path cost 2000     member: epair19a flags=143             ifmaxaddr 0 port 53 priority 128 path cost 2000     member: epair18a flags=143             ifmaxaddr 0 port 48 priority 128 path cost 2000     member: epair16a flags=143             ifmaxaddr 0 port 28 priority 128 path cost 2000     groups: bridge     nd6 options=9 epair16a: flags=8963 metric 0 mtu 1500     description: jail_varnish01     options=8     ether 02:76:32:8e:0e:0a     groups: epair     media: Ethernet 10Gbase-T (10Gbase-T )     status: active     nd6 options=29 epair18a: flags=8963 metric 0 mtu 1500     description: jail_web01     options=8     ether 02:6d:be:b8:36:0a     groups: epair     media: Ethernet 10Gbase-T (10Gbase-T )     status: active     nd6 options=29 epair19a: flags=8963 metric 0 mtu 1500     description: jail_web02     options=8     ether 02:54:fd:77:9a:0a     groups: epair     media: Ethernet 10Gbase-T (10Gbase-T )     status: active     nd6 options=29 epair20a: flags=8963 metric 0 mtu 1500     description: jail_haproxy     options=8     ether 02:f8:58:06:78:0a     groups: epair     media: Ethernet 10Gbase-T (10Gbase-T )     status: active     nd6 options=29 This is on both 13-STABLE and 14-HEAD.