From nobody Fri Jul 23 17:51:42 2021 X-Original-To: freebsd-jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id DDA2E12786E8 for ; Fri, 23 Jul 2021 17:51:51 +0000 (UTC) (envelope-from freebsd@grem.de) Received: from mail.evolve.de (mail.evolve.de [213.239.217.29]) (using TLSv1.3 with cipher TLS_CHACHA20_POLY1305_SHA256 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA512 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mail.evolve.de", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4GWcMW3PLTz3lb3; Fri, 23 Jul 2021 17:51:51 +0000 (UTC) (envelope-from freebsd@grem.de) Received: by mail.evolve.de (OpenSMTPD) with ESMTP id 146e4d9a; Fri, 23 Jul 2021 17:51:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=grem.de; h=date:from:to:cc :subject:message-id:in-reply-to:references:mime-version :content-type:content-transfer-encoding; s=20180501; bh=L67uY/Wp Vm0zQMJRU+/FKJ5Esrk=; b=W75vsmoXTUmbWWqSx4OajTgpuXAF/ad+NUbndThV BRhrW7MDC2d2osnkOjOn/u3C0uGhn9kHrs73HNrypPrsCinxp7SMFaUFxCYIw7mk kSgf2YYOhtxeWMLioim8Xq52P8QHGsGQTkMa1X76dAwY4uovzBA5ui2tq4TsDJ+Z TWRMnMpstTb5FhJqSzPlmONwer2PIbyAOaUsNCdjTta4ZsqmEd86cXMqf2XuU42S 5s82mtjz0l6b0W5e8jR2yy5SsY4/avOgCF6wfnSaDttKpGM/FHg/u2IRG4SKIE6N SxkWctMkjgPAex+8fpzQ1C8qSDeqevdRXn1z/GzqeD6SBg== DomainKey-Signature: a=rsa-sha1; c=nofws; d=grem.de; h=date:from:to:cc :subject:message-id:in-reply-to:references:mime-version :content-type:content-transfer-encoding; q=dns; s=20180501; b=Ku Xu9zYyj7MBxdH8TPjNrVG4B7AhqTp/17VBqkiA37zWPCSNI+GUeLY+C/sPtRrCqj 9A0pxwbXwkPXS8gD82rWAhEOB0BYQHUnUQvBxUkI+1USUxneFICBsUVXJIiRxU94 2ruwFjDzE+29Eh8hd68G+cFvRJ2dDguGMKL8TMmFCFb7yTFduJ8ZdT8kTxTm3MI1 1tEJ1wmgmxDJ+aa9ztDXjFFid/xn1BYJ2Ceq54KeMo9yfAurNyx+OHesP4W0AfW6 nnVyfAnYLjc43k67qGvjeK8V5i4cNJ9wn5yOFXUjlz0UDGUS+JteAkyfxGHNVZPs wHRMtwvGe8BsjphEYHRg== Received: by mail.evolve.de (OpenSMTPD) with ESMTPSA id 067fa54c (TLSv1.3:AEAD-CHACHA20-POLY1305-SHA256:256:NO); Fri, 23 Jul 2021 17:51:45 +0000 (UTC) Date: Fri, 23 Jul 2021 19:51:42 +0200 From: Michael Gmelin To: Jacques Foucry Cc: freebsd-questions@freebsd.org, freebsd-jail@freebsd.org Subject: Re: iocage, vnet jail does not go outside Message-ID: <20210723195142.77b668f1@bsd64.grem.de> In-Reply-To: References: X-Face: $wrgCtfdVw_H9WAY?S&9+/F"!41z'L$uo*WzT8miX?kZ~W~Lr5W7v?j0Sde\mwB&/ypo^}> +a'4xMc^^KroE~+v^&^#[B">soBo1y6(TW6#UZiC]o>C6`ej+i Face: iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAAJFBMVEWJBwe5BQDl LASZU0/LTEWEfHbyj0Txi32+sKrp1Mv944X8/fm1rS+cAAAACXBIWXMAAAsTAAAL EwEAmpwYAAAAB3RJTUUH3wESCxwC7OBhbgAAACFpVFh0Q29tbWVudAAAAAAAQ3Jl YXRlZCB3aXRoIFRoZSBHSU1QbbCXAAAAAghJREFUOMu11DFvEzEUAGCfEhBVFzuq AKkLd0O6VrIQsLXVSZXoWE5N1K3DobBBA9fQpRWc8OkWouaIjedWKiyREOKs+3PY fvalCNjgLVHeF7/3bMtBzV8C/VsQ8tecEgCcDgrzjekwKZ7TwsJZd/ywEKwwP+ZM 8P3drTsAwWn2mpWuDDuYiK1bFs6De0KUUFw0tWxm+D4AIhuuvZqtyWYeO7jQ4Aea 7jUqI+ixhQoHex4WshEvSXdood7stlv4oSuFOC4tqGcr0NjEqXgV4mMJO38nld4+ xKNxRDon7khyKVqY7YR4d+Cg0OMrkWXZOM7YDkEfKiilCn1qYv4mighZiynuHHOA Wq9QJq+BIES7lMFUtcikMnkDGHUoncA+uHgrP0ctIEqfwLHzeSo+eUA66AqzwN6n 2ZHJhw6Qh/PoyC/QENyEyC/AyNjq74Bs+3UH0xYwzDUC4B97HgLocg1QLYgDDO1v f3UX9Y307Ew4AHh67YAFFsxEpkXwpXY3eIgMhAAE3R19L919nNnuD2wlPcDE3UeT L2ytEICQib9BXgS2fU8PrD82ToYO1OEmMSnYTjSqSv9wdC0tPYC+rQRQD9ESnldF CyqfmiYW+tlALt8gH2xrMdC/youbjzPXEun+/ReXsMCDyve3dZc09fn2Oas8oXGc Jj6/fOeK5UmSMPmf/jL+GD8BEj0k/Fn6IO4AAAAASUVORK5CYII= List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4GWcMW3PLTz3lb3 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; TAGGED_RCPT(0.00)[freebsd]; REPLY(-4.00)[] X-ThisMailContainsUnwantedMimeParts: N On Fri, 23 Jul 2021 18:36:25 +0200 Jacques Foucry wrote: > Hello friends, > > I'm turing crazy. > > I made a new jail ,on my hosted system using iocage. > > Here is the config.json file: > > more config.json > { > "allow_mount": 1, > "allow_mount_devfs": 1, > "allow_mount_nullfs": 1, > "allow_mount_procfs": 1, > "allow_mount_tmpfs": 1, > "allow_mount_zfs": 1, > "allow_raw_sockets": 1, > "allow_socket_af": 1, > "allow_sysvipc": 1, > "bpf": 1, > "cloned_release": "13.0-RELEASE", > "defaultrouter": "10.0.10.1", > "defaultrouter6": "auto", > "dhcp": 0, > "host_hostname": "examplejail", > "host_hostuuid": "examplejail", > "ip4_addr": "vnet0|10.0.10.23/24", > "ip6_addr": "vnet0|2a01:4f9:4a:1fd8::23", > "jail_zfs_dataset": "iocage/jails/examplejail/data", > "last_started": "2021-07-23 15:11:28", > "nat": 0, > "release": "13.0-RELEASE-p3", > "vnet": 1, > "vnet0_mac": "b42e999c5bca b42e999c5bcb", > "vnet_default_interface": "auto" > } > > The jail's ifconfig: > > ifconfig > lo0: flags=8049 metric 0 mtu 16384 > options=680003 > inet6 ::1 prefixlen 128 > inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 > inet 127.0.0.1 netmask 0xff000000 > groups: lo > nd6 options=21 > pflog0: flags=0<> metric 0 mtu 33160 > groups: pflog > epair0b: flags=8843 metric 0 > mtu 1500 options=8 > ether b4:2e:99:9c:5b:cb > hwaddr 02:ae:46:07:62:0b > inet 10.0.10.23 netmask 0xffffff00 broadcast 10.0.10.255 > inet6 2a01:4f9:4a:1fd8::23 prefixlen 64 > inet6 fe80::b62e:99ff:fe9c:5bcb%epair0b prefixlen 64 scopeid > 0x3 groups: epair > media: Ethernet 10Gbase-T (10Gbase-T ) > status: active > nd6 options=21 > > The jail's netstat: > > netstat -rn > Routing tables > > Internet: > Destination Gateway Flags Netif Expire > default 10.0.10.1 UGS epair0b > 10.0.10.0/24 link#3 U epair0b > 10.0.10.23 link#3 UHS lo0 > 127.0.0.1 link#1 UH lo0 > > Internet6: > Destination Gateway Flags > Netif Expire ::/96 ::1 > UGRS lo0 default > fe80::1%epair0b UGS epair0b ::1 > link#1 UHS lo0 > ::ffff:0.0.0.0/96 ::1 UGRS > lo0 2a01:4f9:4a:1fd8::/64 link#3 > U epair0b 2a01:4f9:4a:1fd8::23 link#3 > UHS lo0 fe80::/10 ::1 > UGRS lo0 fe80::%lo0/64 > link#1 U lo0 fe80::1%lo0 > link#1 UHS lo0 > fe80::%epair0b/64 link#3 U > epair0b fe80::b62e:99ff:fe9c:5bcb%epair0b link#3 > UHS lo0 ff02::/16 > > On the host, the ifconfig (note thereis a lot of old fashion jails): > > ifconfig > em0: flags=8963 > metric 0 mtu 1500 > options=4810099 > ether b4:2e:99:6a:80:9d inet6 2a01:4f9:4a:1fd8::2 prefixlen 64 > inet6 fe80::b62e:99ff:fe6a:809d%em0 prefixlen 64 scopeid 0x1 > inet6 2a01:4f9:4a:1fd8::5 prefixlen 64 > inet6 2a01:4f9:4a:1fd8::11 prefixlen 64 > inet6 2a01:4f9:4a:1fd8::12 prefixlen 64 > inet6 2a01:4f9:4a:1fd8::15 prefixlen 64 > inet6 2a01:4f9:4a:1fd8::16 prefixlen 64 > inet6 2a01:4f9:4a:1fd8::18 prefixlen 64 > inet6 2a01:4f9:4a:1fd8::19 prefixlen 64 > inet6 2a01:4f9:4a:1fd8::21 prefixlen 64 > inet6 2a01:4f9:4a:1fd8::22 prefixlen 64 > inet6 2a01:4f9:4a:1fd8::25 prefixlen 64 > inet6 2a01:4f9:4a:1fd8::14 prefixlen 64 > inet6 2a01:4f9:4a:1fd8::29 prefixlen 64 > inet6 2a01:4f9:4a:1fd8::17 prefixlen 64 > inet 95.217.83.231 netmask 0xffffffc0 broadcast 95.217.83.255 > media: Ethernet autoselect (1000baseT ) > status: active > nd6 options=21 > lo0: flags=8049 metric 0 mtu 16384 > options=680003 > inet6 ::1 prefixlen 128 > inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 > inet 127.0.0.1 netmask 0xff000000 > inet 127.0.12.1 netmask 0xff000000 > inet 127.0.1.5 netmask 0xffffffff > inet 127.0.1.11 netmask 0xffffffff > inet 127.0.1.12 netmask 0xffffffff > inet 127.0.1.15 netmask 0xffffffff > inet 127.0.1.16 netmask 0xffffffff > inet 127.0.1.18 netmask 0xffffffff > inet 127.0.1.19 netmask 0xffffffff > inet 127.0.1.21 netmask 0xffffffff > inet 127.0.1.22 netmask 0xffffffff > inet 127.0.1.25 netmask 0xffffffff > inet 127.0.1.14 netmask 0xffffffff > inet 127.0.1.29 netmask 0xffffffff > inet 127.0.1.17 netmask 0xffffffff > groups: lo > nd6 options=21 > lo1: flags=8049 metric 0 mtu 16384 > options=680003 > inet 192.168.12.1 netmask 0xffffff00 > inet 192.168.12.5 netmask 0xffffffff > inet 192.168.12.11 netmask 0xffffff00 > inet 192.168.12.12 netmask 0xffffff00 > inet 192.168.12.15 netmask 0xffffff00 > inet 192.168.12.16 netmask 0xffffff00 > inet 192.168.12.18 netmask 0xffffff00 > inet 192.168.12.19 netmask 0xffffff00 > inet 192.168.12.21 netmask 0xffffff00 > inet 192.168.12.22 netmask 0xffffff00 > inet 192.168.12.25 netmask 0xffffff00 > inet 192.168.12.14 netmask 0xffffff00 > inet 192.168.12.29 netmask 0xffffff00 > inet 192.168.12.17 netmask 0xffffff00 > groups: lo > nd6 options=29 > pflog0: flags=100 metric 0 mtu 33160 > groups: pflog > bridge0: flags=8843 metric 0 > mtu 1500 description: jails-bridge > ether 58:9c:fc:10:ed:66 > inet 10.0.10.1 netmask 0xffffff00 broadcast 10.0.10.255 > id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 > maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 > root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 > member: vnet0.655 > flags=143 ifmaxaddr 0 port 6 > priority 128 path cost 2000 member: em0 > flags=143 ifmaxaddr 0 port 1 > priority 128 path cost 20000 groups: bridge > nd6 options=9 > vnet0.655: flags=8943 > metric 0 mtu 1500 description: associated with jail: examplejail as > nic: epair0b options=8 > ether b4:2e:99:9c:5b:ca > hwaddr 02:ae:46:07:62:0a > groups: epair > media: Ethernet 10Gbase-T (10Gbase-T ) > status: active > nd6 options=29 > > And host's netstat (again with many old fashion jail): > > netstat -rn > Routing tables > > Internet: > Destination Gateway Flags Netif Expire > default 95.217.83.193 UGS em0 > 10.0.10.0/24 link#5 U bridge0 > 10.0.10.1 link#5 UHS lo0 > 95.217.83.192/26 link#1 U em0 > 95.217.83.231 link#1 UHS lo0 > 127.0.0.1 link#2 UH lo0 > 127.0.1.5 link#2 UH lo0 > 127.0.1.11 link#2 UH lo0 > 127.0.1.12 link#2 UH lo0 > 127.0.1.14 link#2 UH lo0 > 127.0.1.15 link#2 UH lo0 > 127.0.1.16 link#2 UH lo0 > 127.0.1.17 link#2 UH lo0 > 127.0.1.18 link#2 UH lo0 > 127.0.1.19 link#2 UH lo0 > 127.0.1.21 link#2 UH lo0 > 127.0.1.22 link#2 UH lo0 > 127.0.1.25 link#2 UH lo0 > 127.0.1.29 link#2 UH lo0 > 127.0.12.1 link#2 UH lo0 > 192.168.12.1 link#3 UH lo1 > 192.168.12.5 link#3 UH lo1 > 192.168.12.11 link#3 UH lo1 > 192.168.12.12 link#3 UH lo1 > 192.168.12.14 link#3 UH lo1 > 192.168.12.15 link#3 UH lo1 > 192.168.12.16 link#3 UH lo1 > 192.168.12.17 link#3 UH lo1 > 192.168.12.18 link#3 UH lo1 > 192.168.12.19 link#3 UH lo1 > 192.168.12.21 link#3 UH lo1 > 192.168.12.22 link#3 UH lo1 > 192.168.12.25 link#3 UH lo1 > 192.168.12.29 link#3 UH lo1 > > Internet6: > Destination Gateway Flags > Netif Expire ::/96 ::1 > UGRS lo0 default > fe80::1%em0 UGS em0 ::1 > link#2 UHS lo0 > ::ffff:0.0.0.0/96 ::1 UGRS > lo0 2a01:4f9:4a:1fd8::/64 link#1 > U em0 2a01:4f9:4a:1fd8::2 link#1 > UHS lo0 2a01:4f9:4a:1fd8::5 > link#1 UHS lo0 2a01:4f9:4a:1fd8::11 > link#1 UHS lo0 > 2a01:4f9:4a:1fd8::12 link#1 UHS > lo0 2a01:4f9:4a:1fd8::14 link#1 > UHS lo0 2a01:4f9:4a:1fd8::15 link#1 > UHS lo0 2a01:4f9:4a:1fd8::16 > link#1 UHS lo0 2a01:4f9:4a:1fd8::17 > link#1 UHS lo0 > 2a01:4f9:4a:1fd8::18 link#1 UHS > lo0 2a01:4f9:4a:1fd8::19 link#1 > UHS lo0 2a01:4f9:4a:1fd8::21 link#1 > UHS lo0 2a01:4f9:4a:1fd8::22 > link#1 UHS lo0 2a01:4f9:4a:1fd8::25 > link#1 UHS lo0 > 2a01:4f9:4a:1fd8::29 link#1 UHS > lo0 fe80::/10 ::1 > UGRS lo0 fe80::%em0/64 link#1 > U em0 fe80::b62e:99ff:fe6a:809d%em0 > link#1 UHS lo0 fe80::%lo0/64 > link#2 U lo0 fe80::1%lo0 > link#2 UHS lo0 > ff02::/16 ::1 UGRS > lo0 > > The bridge0 had the em0 and vnet0:655 interfaces. > > From the jail in can ping oustside world: > > ping google.ca > PING6(56=40+8+8 bytes) 2a01:4f9:4a:1fd8::23 --> > 2a00:1450:400f:803::2003 16 bytes from 2a00:1450:400f:803::2003, > icmp_seq=0 hlim=118 time=7.927 ms 16 bytes from > 2a00:1450:400f:803::2003, icmp_seq=1 hlim=118 time=7.800 ms 16 bytes > from 2a00:1450:400f:803::2003, icmp_seq=2 hlim=118 time=7.798 ms ^C > --- google.ca ping6 statistics --- > 3 packets transmitted, 3 packets received, 0.0% packet loss > round-trip min/avg/max/std-dev = 7.798/7.842/7.927/0.061 ms > > The problem is, I cannot ssh to an external computer (for example, my > nextcloud hosted at home): > > ssh -vvv nextcloud.foucry.net -p2250 > OpenSSH_7.9p1, OpenSSL 1.1.1k-freebsd 25 Mar 2021 > debug1: Reading configuration data /etc/ssh/ssh_config > debug2: resolving "nextcloud.foucry.net" port 2250 > debug2: ssh_connect_direct > debug1: Connecting to nextcloud.foucry.net > [2a01:e0a:434:44e0:ff:60ff:feba:b582] port 2250. debug1: connect to > address 2a01:e0a:434:44e0:ff:60ff:feba:b582 port 2250: Operation > timed out debug1: Connecting to nextcloud.foucry.net [82.65.174.130] > port 2250. debug1: connect to address 82.65.174.130 port 2250: > Operation timed out ssh: connect to host nextcloud.foucry.net port > 2250: Operation timed out > > What's look strange (for me) is the traceroute (using ipv4): > > traceroute nextcloud.foucry.net > traceroute to nextcloud.foucry.net (82.65.174.130), 64 hops max, 40 > byte packets 1 10.0.10.1 (10.0.10.1) 0.086 ms 0.051 ms 0.037 ms > 2 static.193.83.217.95.clients.your-server.de (95.217.83.193) > 0.451 ms 0.571 ms 0.392 ms 3 core32.hel1.hetzner.com > (213.239.252.97) 11.621 ms core31.hel1.hetzner.com (213.239.252.93) > 1.812 ms core32.hel1.hetzner.com (213.239.252.97) 2.793 ms > 4 core9.fra.hetzner.com (213.239.224.166) 21.295 ms > core8.fra.hetzner.com (213.239.224.149) 20.730 ms > core9.fra.hetzner.com (213.239.224.170) 20.333 ms > 5 core4.fra.hetzner.com (213.239.245.85) 28.499 ms > core4.fra.hetzner.com (213.239.224.177) 20.507 ms 22.850 ms > 6 * * * > 7 * * * > 8 * * * > 9 * * * > 10 * * * > 11 * * * > 12 * * * > 13 *^C > > > Look's like something wrong on the way, but I could connect on the > same host form any other jails. > > > There is for me a mysterious behaviiors that I can't understand. > > Any help will be appreciate. > > Thanks for reading me, and the time your spend on my problem. You need to enable some sort of NAT at your end, e.g. using pf. Traffic is leaving your host on a private IP. -m -- Michael Gmelin