UDP connections from NAT'ed jails

[Peter Ludikovsky]

Hi,

I'm experimenting with jails in preparation for moving my home server
from Linux to FreeBSD. I'm doing this from within a VirtualBox VM, since
it's easier to revert to a previous state in case I break something.

My biggest issue ATM is that my first jail can't resolve any host. TCP
and ICMP packets pass without issue, but DNS requests time out. I
checked with tcpdump on both the outside interface of the VM and of the
host, neither show any DNS requests. Both hosts use 9.9.9.10 as the DNS
server in /etc/resolv.conf.

On the host:

    [peter at doctor ~]$ ifconfig -a
    em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
        ether 08:00:27:8f:47:bc
        hwaddr 08:00:27:8f:47:bc
        inet 10.0.2.15 netmask 0xffffff00 broadcast 10.0.2.255 
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
    lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128 
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 
        inet 127.0.0.1 netmask 0xff000000 
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: lo 
    lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet 192.168.5.1 netmask 0xffffff00 
        inet 192.168.5.3 netmask 0xffffffff 
        inet 192.168.5.4 netmask 0xffffffff 
        inet 192.168.5.5 netmask 0xffffffff 
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        groups: lo 
    [peter at doctor ~]$ cat /usr/local/etc/ezjail/bind9 
    # To specify the start up order of your ezjails, use these lines to
    # create a Jail dependency tree. See rcorder(8) for more details.
    #
    # PROVIDE: standard_ezjail
    # REQUIRE: 
    # BEFORE: 
    #

    export jail_bind9_hostname="bind9"
    export jail_bind9_ip="192.168.5.3"
    export jail_bind9_rootdir="/usr/jails/bind9"
    export jail_bind9_exec_start="/bin/sh /etc/rc"
    export jail_bind9_exec_stop=""
    export jail_bind9_mount_enable="YES"
    export jail_bind9_devfs_enable="YES"
    export jail_bind9_devfs_ruleset="devfsrules_jail"
    export jail_bind9_procfs_enable="YES"
    export jail_bind9_fdescfs_enable="YES"
    export jail_bind9_image=""
    export jail_bind9_imagetype="zfs"
    export jail_bind9_attachparams=""
    export jail_bind9_attachblocking=""
    export jail_bind9_forceblocking=""
    export jail_bind9_zfs_datasets=""
    export jail_bind9_cpuset=""
    export jail_bind9_fib=""
    export jail_bind9_parentzfs="data/jails"
    export jail_bind9_parameters="allow.raw_sockets=1"
    export jail_bind9_post_start_script=""
    export jail_bind9_retention_policy=""
    [peter at doctor ~]$ nc -z -w 1 pkg.freebsd.org 80; echo $?
    Connection to pkg.freebsd.org 80 port [tcp/http] succeeded!
    0
    [peter at doctor ~]$ nc -z -w 1 149.20.1.201 80 ; echo $?
    Connection to 149.20.1.201 80 port [tcp/http] succeeded!
    0
    [peter at doctor ~]$ cat /etc/pf.conf 
    IP_PUB="10.0.2.15"
    IP_JAIL="192.168.5.2"
    NET_JAIL="192.168.5.0/24"
    scrub in all
    set skip on lo
    nat pass on em0 from $NET_JAIL to any -> $IP_PUB
    pass out keep state
    [peter at doctor ~]$ sudo pfctl -sn
    nat pass on em0 inet from 192.168.5.0/24 to any -> 10.0.2.15

In the jail:

    root at bind9:~ # ifconfig -a
    em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
        ether 08:00:27:8f:47:bc
        hwaddr 08:00:27:8f:47:bc
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
    lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        groups: lo 
    lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet 192.168.5.3 netmask 0xffffffff 
        groups: lo 
    root at bind9:~ # netstat -r
    netstat: kvm not available: /dev/mem: No such file or directory
    Routing tables

    Internet:
    Destination        Gateway            Flags     Netif Expire
    192.168.5.3        link#3             UH          lo1
    root at bind9:~ # cat /etc/resolv.conf 
    server 10.1.9.253
    root at bind9:~ # host pkg.freebsd.org
    ;; connection timed out; no servers could be reached
    root at bind9:~ # nc -z -w 1 pkg.freebsd.org 80 ; echo $?
    nc: getaddrinfo: hostname nor servname provided, or not known
    1
    root at bind9:~ # nc -z -w 1 149.20.1.201 80 ; echo $?
    Connection to 149.20.1.201 80 port [tcp/http] succeeded!
    0
    root at bind9:~ # ping -c3 pkg.freebsd.org
    ping: cannot resolve pkg.freebsd.org: Host name lookup failure
    root at bind9:~ # ping -c3 149.20.1.201
    PING 149.20.1.201 (149.20.1.201): 56 data bytes
    64 bytes from 149.20.1.201: icmp_seq=0 ttl=63 time=165.686 ms
    64 bytes from 149.20.1.201: icmp_seq=1 ttl=63 time=164.283 ms
    64 bytes from 149.20.1.201: icmp_seq=2 ttl=63 time=165.578 ms

    --- 149.20.1.201 ping statistics ---
    3 packets transmitted, 3 packets received, 0.0% packet loss
    round-trip min/avg/max/stddev = 164.283/165.182/165.686/0.637 ms

Anyone got a pointer on what's going wrong here?

Regards,
/peter