Re: iocage, vnet jail does not go outside
- Reply: Jacques Foucry : "Re: iocage, vnet jail does not go outside"
- In reply to: Jacques Foucry : "iocage, vnet jail does not go outside"
- Go to: [ bottom of page ] [ top of archives ] [ this month ]
Date: Fri, 23 Jul 2021 17:51:42 UTC
On Fri, 23 Jul 2021 18:36:25 +0200
Jacques Foucry <jacques+freebsd@foucry.net> wrote:
> Hello friends,
>
> I'm turing crazy.
>
> I made a new jail ,on my hosted system using iocage.
>
> Here is the config.json file:
>
> more config.json
> {
> "allow_mount": 1,
> "allow_mount_devfs": 1,
> "allow_mount_nullfs": 1,
> "allow_mount_procfs": 1,
> "allow_mount_tmpfs": 1,
> "allow_mount_zfs": 1,
> "allow_raw_sockets": 1,
> "allow_socket_af": 1,
> "allow_sysvipc": 1,
> "bpf": 1,
> "cloned_release": "13.0-RELEASE",
> "defaultrouter": "10.0.10.1",
> "defaultrouter6": "auto",
> "dhcp": 0,
> "host_hostname": "examplejail",
> "host_hostuuid": "examplejail",
> "ip4_addr": "vnet0|10.0.10.23/24",
> "ip6_addr": "vnet0|2a01:4f9:4a:1fd8::23",
> "jail_zfs_dataset": "iocage/jails/examplejail/data",
> "last_started": "2021-07-23 15:11:28",
> "nat": 0,
> "release": "13.0-RELEASE-p3",
> "vnet": 1,
> "vnet0_mac": "b42e999c5bca b42e999c5bcb",
> "vnet_default_interface": "auto"
> }
>
> The jail's ifconfig:
>
> ifconfig
> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
> options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
> inet6 ::1 prefixlen 128
> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
> inet 127.0.0.1 netmask 0xff000000
> groups: lo
> nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> pflog0: flags=0<> metric 0 mtu 33160
> groups: pflog
> epair0b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0
> mtu 1500 options=8<VLAN_MTU>
> ether b4:2e:99:9c:5b:cb
> hwaddr 02:ae:46:07:62:0b
> inet 10.0.10.23 netmask 0xffffff00 broadcast 10.0.10.255
> inet6 2a01:4f9:4a:1fd8::23 prefixlen 64
> inet6 fe80::b62e:99ff:fe9c:5bcb%epair0b prefixlen 64 scopeid
> 0x3 groups: epair
> media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
> status: active
> nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
>
> The jail's netstat:
>
> netstat -rn
> Routing tables
>
> Internet:
> Destination Gateway Flags Netif Expire
> default 10.0.10.1 UGS epair0b
> 10.0.10.0/24 link#3 U epair0b
> 10.0.10.23 link#3 UHS lo0
> 127.0.0.1 link#1 UH lo0
>
> Internet6:
> Destination Gateway Flags
> Netif Expire ::/96 ::1
> UGRS lo0 default
> fe80::1%epair0b UGS epair0b ::1
> link#1 UHS lo0
> ::ffff:0.0.0.0/96 ::1 UGRS
> lo0 2a01:4f9:4a:1fd8::/64 link#3
> U epair0b 2a01:4f9:4a:1fd8::23 link#3
> UHS lo0 fe80::/10 ::1
> UGRS lo0 fe80::%lo0/64
> link#1 U lo0 fe80::1%lo0
> link#1 UHS lo0
> fe80::%epair0b/64 link#3 U
> epair0b fe80::b62e:99ff:fe9c:5bcb%epair0b link#3
> UHS lo0 ff02::/16
>
> On the host, the ifconfig (note thereis a lot of old fashion jails):
>
> ifconfig
> em0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST>
> metric 0 mtu 1500
> options=4810099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER,NOMAP>
> ether b4:2e:99:6a:80:9d inet6 2a01:4f9:4a:1fd8::2 prefixlen 64
> inet6 fe80::b62e:99ff:fe6a:809d%em0 prefixlen 64 scopeid 0x1
> inet6 2a01:4f9:4a:1fd8::5 prefixlen 64
> inet6 2a01:4f9:4a:1fd8::11 prefixlen 64
> inet6 2a01:4f9:4a:1fd8::12 prefixlen 64
> inet6 2a01:4f9:4a:1fd8::15 prefixlen 64
> inet6 2a01:4f9:4a:1fd8::16 prefixlen 64
> inet6 2a01:4f9:4a:1fd8::18 prefixlen 64
> inet6 2a01:4f9:4a:1fd8::19 prefixlen 64
> inet6 2a01:4f9:4a:1fd8::21 prefixlen 64
> inet6 2a01:4f9:4a:1fd8::22 prefixlen 64
> inet6 2a01:4f9:4a:1fd8::25 prefixlen 64
> inet6 2a01:4f9:4a:1fd8::14 prefixlen 64
> inet6 2a01:4f9:4a:1fd8::29 prefixlen 64
> inet6 2a01:4f9:4a:1fd8::17 prefixlen 64
> inet 95.217.83.231 netmask 0xffffffc0 broadcast 95.217.83.255
> media: Ethernet autoselect (1000baseT <full-duplex>)
> status: active
> nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
> options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
> inet6 ::1 prefixlen 128
> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
> inet 127.0.0.1 netmask 0xff000000
> inet 127.0.12.1 netmask 0xff000000
> inet 127.0.1.5 netmask 0xffffffff
> inet 127.0.1.11 netmask 0xffffffff
> inet 127.0.1.12 netmask 0xffffffff
> inet 127.0.1.15 netmask 0xffffffff
> inet 127.0.1.16 netmask 0xffffffff
> inet 127.0.1.18 netmask 0xffffffff
> inet 127.0.1.19 netmask 0xffffffff
> inet 127.0.1.21 netmask 0xffffffff
> inet 127.0.1.22 netmask 0xffffffff
> inet 127.0.1.25 netmask 0xffffffff
> inet 127.0.1.14 netmask 0xffffffff
> inet 127.0.1.29 netmask 0xffffffff
> inet 127.0.1.17 netmask 0xffffffff
> groups: lo
> nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
> options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
> inet 192.168.12.1 netmask 0xffffff00
> inet 192.168.12.5 netmask 0xffffffff
> inet 192.168.12.11 netmask 0xffffff00
> inet 192.168.12.12 netmask 0xffffff00
> inet 192.168.12.15 netmask 0xffffff00
> inet 192.168.12.16 netmask 0xffffff00
> inet 192.168.12.18 netmask 0xffffff00
> inet 192.168.12.19 netmask 0xffffff00
> inet 192.168.12.21 netmask 0xffffff00
> inet 192.168.12.22 netmask 0xffffff00
> inet 192.168.12.25 netmask 0xffffff00
> inet 192.168.12.14 netmask 0xffffff00
> inet 192.168.12.29 netmask 0xffffff00
> inet 192.168.12.17 netmask 0xffffff00
> groups: lo
> nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> pflog0: flags=100<PROMISC> metric 0 mtu 33160
> groups: pflog
> bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0
> mtu 1500 description: jails-bridge
> ether 58:9c:fc:10:ed:66
> inet 10.0.10.1 netmask 0xffffff00 broadcast 10.0.10.255
> id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
> maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
> root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
> member: vnet0.655
> flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 6
> priority 128 path cost 2000 member: em0
> flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 1
> priority 128 path cost 20000 groups: bridge
> nd6 options=9<PERFORMNUD,IFDISABLED>
> vnet0.655: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST>
> metric 0 mtu 1500 description: associated with jail: examplejail as
> nic: epair0b options=8<VLAN_MTU>
> ether b4:2e:99:9c:5b:ca
> hwaddr 02:ae:46:07:62:0a
> groups: epair
> media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
> status: active
> nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
>
> And host's netstat (again with many old fashion jail):
>
> netstat -rn
> Routing tables
>
> Internet:
> Destination Gateway Flags Netif Expire
> default 95.217.83.193 UGS em0
> 10.0.10.0/24 link#5 U bridge0
> 10.0.10.1 link#5 UHS lo0
> 95.217.83.192/26 link#1 U em0
> 95.217.83.231 link#1 UHS lo0
> 127.0.0.1 link#2 UH lo0
> 127.0.1.5 link#2 UH lo0
> 127.0.1.11 link#2 UH lo0
> 127.0.1.12 link#2 UH lo0
> 127.0.1.14 link#2 UH lo0
> 127.0.1.15 link#2 UH lo0
> 127.0.1.16 link#2 UH lo0
> 127.0.1.17 link#2 UH lo0
> 127.0.1.18 link#2 UH lo0
> 127.0.1.19 link#2 UH lo0
> 127.0.1.21 link#2 UH lo0
> 127.0.1.22 link#2 UH lo0
> 127.0.1.25 link#2 UH lo0
> 127.0.1.29 link#2 UH lo0
> 127.0.12.1 link#2 UH lo0
> 192.168.12.1 link#3 UH lo1
> 192.168.12.5 link#3 UH lo1
> 192.168.12.11 link#3 UH lo1
> 192.168.12.12 link#3 UH lo1
> 192.168.12.14 link#3 UH lo1
> 192.168.12.15 link#3 UH lo1
> 192.168.12.16 link#3 UH lo1
> 192.168.12.17 link#3 UH lo1
> 192.168.12.18 link#3 UH lo1
> 192.168.12.19 link#3 UH lo1
> 192.168.12.21 link#3 UH lo1
> 192.168.12.22 link#3 UH lo1
> 192.168.12.25 link#3 UH lo1
> 192.168.12.29 link#3 UH lo1
>
> Internet6:
> Destination Gateway Flags
> Netif Expire ::/96 ::1
> UGRS lo0 default
> fe80::1%em0 UGS em0 ::1
> link#2 UHS lo0
> ::ffff:0.0.0.0/96 ::1 UGRS
> lo0 2a01:4f9:4a:1fd8::/64 link#1
> U em0 2a01:4f9:4a:1fd8::2 link#1
> UHS lo0 2a01:4f9:4a:1fd8::5
> link#1 UHS lo0 2a01:4f9:4a:1fd8::11
> link#1 UHS lo0
> 2a01:4f9:4a:1fd8::12 link#1 UHS
> lo0 2a01:4f9:4a:1fd8::14 link#1
> UHS lo0 2a01:4f9:4a:1fd8::15 link#1
> UHS lo0 2a01:4f9:4a:1fd8::16
> link#1 UHS lo0 2a01:4f9:4a:1fd8::17
> link#1 UHS lo0
> 2a01:4f9:4a:1fd8::18 link#1 UHS
> lo0 2a01:4f9:4a:1fd8::19 link#1
> UHS lo0 2a01:4f9:4a:1fd8::21 link#1
> UHS lo0 2a01:4f9:4a:1fd8::22
> link#1 UHS lo0 2a01:4f9:4a:1fd8::25
> link#1 UHS lo0
> 2a01:4f9:4a:1fd8::29 link#1 UHS
> lo0 fe80::/10 ::1
> UGRS lo0 fe80::%em0/64 link#1
> U em0 fe80::b62e:99ff:fe6a:809d%em0
> link#1 UHS lo0 fe80::%lo0/64
> link#2 U lo0 fe80::1%lo0
> link#2 UHS lo0
> ff02::/16 ::1 UGRS
> lo0
>
> The bridge0 had the em0 and vnet0:655 interfaces.
>
> From the jail in can ping oustside world:
>
> ping google.ca
> PING6(56=40+8+8 bytes) 2a01:4f9:4a:1fd8::23 -->
> 2a00:1450:400f:803::2003 16 bytes from 2a00:1450:400f:803::2003,
> icmp_seq=0 hlim=118 time=7.927 ms 16 bytes from
> 2a00:1450:400f:803::2003, icmp_seq=1 hlim=118 time=7.800 ms 16 bytes
> from 2a00:1450:400f:803::2003, icmp_seq=2 hlim=118 time=7.798 ms ^C
> --- google.ca ping6 statistics ---
> 3 packets transmitted, 3 packets received, 0.0% packet loss
> round-trip min/avg/max/std-dev = 7.798/7.842/7.927/0.061 ms
>
> The problem is, I cannot ssh to an external computer (for example, my
> nextcloud hosted at home):
>
> ssh -vvv nextcloud.foucry.net -p2250
> OpenSSH_7.9p1, OpenSSL 1.1.1k-freebsd 25 Mar 2021
> debug1: Reading configuration data /etc/ssh/ssh_config
> debug2: resolving "nextcloud.foucry.net" port 2250
> debug2: ssh_connect_direct
> debug1: Connecting to nextcloud.foucry.net
> [2a01:e0a:434:44e0:ff:60ff:feba:b582] port 2250. debug1: connect to
> address 2a01:e0a:434:44e0:ff:60ff:feba:b582 port 2250: Operation
> timed out debug1: Connecting to nextcloud.foucry.net [82.65.174.130]
> port 2250. debug1: connect to address 82.65.174.130 port 2250:
> Operation timed out ssh: connect to host nextcloud.foucry.net port
> 2250: Operation timed out
>
> What's look strange (for me) is the traceroute (using ipv4):
>
> traceroute nextcloud.foucry.net
> traceroute to nextcloud.foucry.net (82.65.174.130), 64 hops max, 40
> byte packets 1 10.0.10.1 (10.0.10.1) 0.086 ms 0.051 ms 0.037 ms
> 2 static.193.83.217.95.clients.your-server.de (95.217.83.193)
> 0.451 ms 0.571 ms 0.392 ms 3 core32.hel1.hetzner.com
> (213.239.252.97) 11.621 ms core31.hel1.hetzner.com (213.239.252.93)
> 1.812 ms core32.hel1.hetzner.com (213.239.252.97) 2.793 ms
> 4 core9.fra.hetzner.com (213.239.224.166) 21.295 ms
> core8.fra.hetzner.com (213.239.224.149) 20.730 ms
> core9.fra.hetzner.com (213.239.224.170) 20.333 ms
> 5 core4.fra.hetzner.com (213.239.245.85) 28.499 ms
> core4.fra.hetzner.com (213.239.224.177) 20.507 ms 22.850 ms
> 6 * * *
> 7 * * *
> 8 * * *
> 9 * * *
> 10 * * *
> 11 * * *
> 12 * * *
> 13 *^C
>
>
> Look's like something wrong on the way, but I could connect on the
> same host form any other jails.
>
>
> There is for me a mysterious behaviiors that I can't understand.
>
> Any help will be appreciate.
>
> Thanks for reading me, and the time your spend on my problem.
You need to enable some sort of NAT at your end, e.g. using pf. Traffic
is leaving your host on a private IP.
-m
--
Michael Gmelin