IPFW NAT in VNET jail

[Giacomo Olgeni]

On Fri, 22 Feb 2019, Giacomo Olgeni wrote:

> 00100      0        0 nat 1 ip from any to any recv epair0b
> 00200     74     4080 nat 1 ip from any to any xmit epair0b
> 00300      0        0 check-state :default
> 00400      6      360 allow tcp from any to any out xmit epair0b setup keep-state :default
> 00500      0        0 allow udp from any to any out xmit epair0b setup
> 65535 113149 11125118 allow ip from any to any

More testing with a know good configuration, that I got by setting
a few things in rc.conf.

On the VNET jail, which doesn't work, I have:

rc.conf
=======

defaultrouter="10.64.23.33"
firewall_enable="YES"
firewall_nat_enable="YES"
firewall_nat_interface="epair0b"
firewall_type="open"

Rules
=====

00050  18   860 nat 123 ip4 from any to any via epair0b
00100   0     0 allow ip from any to any via lo0
00200   0     0 deny ip from any to 127.0.0.0/8
00300   0     0 deny ip from 127.0.0.0/8 to any
00400   0     0 deny ip from any to ::1
00500   0     0 deny ip from ::1 to any
00600   1    76 allow ipv6-icmp from :: to ff02::/16
00700   0     0 allow ipv6-icmp from fe80::/10 to fe80::/10
00800   0     0 allow ipv6-icmp from fe80::/10 to ff02::/16
00900   0     0 allow ipv6-icmp from any to any icmp6types 1
01000   0     0 allow ipv6-icmp from any to any icmp6types 2,135,136
65000 175 16234 allow ip from any to any
65535  15  1209 allow ip from any to any

On a bhyve VM, same host, same bridge(4) interface, I have...

rc.conf
=======

defaultrouter="10.64.23.33"
firewall_enable="YES"
firewall_nat_enable="YES"
firewall_nat_interface="vtnet1"
firewall_type="open"
ifconfig_vtnet0="inet 192.168.3.202/24"
ifconfig_vtnet1="inet 10.64.23.34/28"

Rules
=====

00050 10  839 nat 123 ip4 from any to any via vtnet1
00100 64 6304 allow ip from any to any via lo0
00200  0    0 deny ip from any to 127.0.0.0/8
00300  0    0 deny ip from 127.0.0.0/8 to any
00400  0    0 deny ip from any to ::1
00500  0    0 deny ip from ::1 to any
00600  0    0 allow ipv6-icmp from :: to ff02::/16
00700  0    0 allow ipv6-icmp from fe80::/10 to fe80::/10
00800  0    0 allow ipv6-icmp from fe80::/10 to ff02::/16
00900  0    0 allow ipv6-icmp from any to any icmp6types 1
01000  0    0 allow ipv6-icmp from any to any icmp6types 2,135,136
65000 34 2767 allow ip from any to any
65535  0    0 allow ip from any to any

And the NAT works correctly. So, there's apparently something going on
with in-kernel NAT and VNET, but I don't have the faintest clue where
to look...

Problem not solved but cheated around :)