NAT before IPSEC - reply packets stuck at enc0

Mon Jul 24 15:14:17 UTC 2017

Am 24.07.2017 um 13:18 schrieb Andrey V. Elsukov:
>
> Ok, let's try to debug the problem. Please, use 11.1-RC, it has
> significantly changed IPsec stack.
>
> Apply attached patch to if_enc(4), it makes if_enc a bit useful for
> debugging your problem. You need to rebuild and reinstall
> sys/modules/if_enc.
>
> Now enable verbose BPF logging:
> net.enc.out.ipsec_bpf_mask=3
> net.enc.in.ipsec_bpf_mask=3
>
> According your tcpdump output, you need to set
> net.enc.out.ipsec_filter_mask=2
>
> Show what you will see in the `tcpdump -nvi enc0` with such config
> options. Also, show what you have in the `sysctl  net.inet.ip.fw` and
> `ipfw show` output.
>
Great! The guys from OPNsense built me a custom 11.1 kernel with your patch.

Here's one packet on enc0:

root at PB-FW1-FRA:~ # tcpdump -vni enc0
tcpdump: listening on enc0, link-type ENC (OpenBSD encapsulated IP), 
capture size 262144 bytes
17:07:41.769313 (authentic,confidential): SPI 0x90b7ba72: IP (tos 0x0, 
ttl 63, id 27752, offset 0, flags [none], proto ICMP (1), length 28, bad 
cksum b72d (->b82d)!)
     10.26.1.1 > 10.24.66.25: ICMP echo request, id 41163, seq 28416, 
length 8
17:07:41.777223 (authentic,confidential): SPI 0xcaae18f8: IP (tos 0x0, 
ttl 58, id 44180, offset 0, flags [none], proto IPIP (4), length 48)
     81.24.74.3 > 213.244.192.191: IP (tos 0x0, ttl 63, id 46327, offset 
0, flags [none], proto ICMP (1), length 28)
     10.24.66.25 > 10.26.1.1: ICMP echo reply, id 41163, seq 28416, length 8
17:07:41.777240 (authentic,confidential): SPI 0xcaae18f8: IP (tos 0x0, 
ttl 63, id 46327, offset 0, flags [none], proto ICMP (1), length 28)
     10.26.1.1 > 10.26.1.1: ICMP echo reply, id 33347, seq 28416, length 8
17:07:41.846588 (authentic,confidential): SPI 0x90b7ba72: IP (tos 0x0, 
ttl 63, id 61607, offset 0, flags [none], proto ICMP (1), length 28, bad 
cksum 32ee (->33ee)!)
     10.26.1.1 > 10.24.66.25: ICMP echo request, id 45562, seq 58116, 
length 8
17:07:41.854692 (authentic,confidential): SPI 0xcaae18f8: IP (tos 0x0, 
ttl 58, id 44196, offset 0, flags [none], proto IPIP (4), length 48)
     81.24.74.3 > 213.244.192.191: IP (tos 0x0, ttl 63, id 46335, offset 
0, flags [none], proto ICMP (1), length 28)
     10.24.66.25 > 10.26.1.1: ICMP echo reply, id 45562, seq 58116, length 8
17:07:41.854706 (authentic,confidential): SPI 0xcaae18f8: IP (tos 0x0, 
ttl 63, id 46335, offset 0, flags [none], proto ICMP (1), length 28)
     10.26.1.1 > 10.26.1.1: ICMP echo reply, id 40754, seq 58116, length 8

ipfw show:

root at PB-FW1-FRA:~ # ipfw show
00100     0       0 allow pfsync from any to any
00110     0       0 allow carp from any to any
00120     0       0 allow ip from any to any layer2 mac-type 0x0806,0x8035
00130     0       0 allow ip from any to any layer2 mac-type 0x888e,0x88c7
00140     0       0 allow ip from any to any layer2 mac-type 0x8863,0x8864
00150     0       0 deny ip from any to any layer2 not mac-type 
0x0800,0x86dd
00179   410   11480 nat 1 log ip from 10.26.2.0/24 to 10.24.66.0/24
00179   414   11816 nat 1 log ip from 10.24.66.0/24 to 10.26.1.1 in recv 
enc0
00200     0       0 skipto 60000 ip6 from ::1 to any
00201    44   41006 skipto 60000 ip4 from 127.0.0.0/8 to any
00202     0       0 skipto 60000 ip6 from any to ::1
00203     0       0 skipto 60000 ip4 from any to 127.0.0.0/8
01002     0       0 skipto 60000 udp from any to 10.26.1.1 dst-port 53 
keep-state
01002     4     336 skipto 60000 ip from any to { 255.255.255.255 or 
10.26.1.1 } in
01002   463   14672 skipto 60000 ip from { 255.255.255.255 or 10.26.1.1 
} to any out
01002     0       0 skipto 60000 icmp from { 255.255.255.255 or 
10.26.1.1 } to any out icmptypes 0
01002     0       0 skipto 60000 icmp from any to { 255.255.255.255 or 
10.26.1.1 } in icmptypes 8
06000  5131 4476281 skipto 60000 tcp from any to any out
06199 10768 1914882 skipto 60000 ip from any to any
30000     0       0 count ip from any to any
60000     0       0 return ip from any to any
60001     0       0 queue 10000 tcp from any to 10.24.66.0/24 via enc0
65533 16410 6447177 allow ip from any to any
65534     0       0 deny ip from any to any
65535     0       0 deny ip from any to any

sysctl:

net.enc.out.ipsec_bpf_mask: 3
net.enc.out.ipsec_filter_mask: 1
net.enc.in.ipsec_bpf_mask: 3
net.enc.in.ipsec_filter_mask: 2
net.enc.out.ipsec_bpf_mask: 3
net.enc.out.ipsec_filter_mask: 1
net.enc.in.ipsec_bpf_mask: 3
net.enc.in.ipsec_filter_mask: 2

root at PB-FW1-FRA:~ # sysctl  net.inet.ip.fw
net.inet.ip.fw.dyn_keep_states: 0
net.inet.ip.fw.dyn_keepalive: 1
net.inet.ip.fw.dyn_short_lifetime: 5
net.inet.ip.fw.dyn_udp_lifetime: 10
net.inet.ip.fw.dyn_rst_lifetime: 1
net.inet.ip.fw.dyn_fin_lifetime: 1
net.inet.ip.fw.dyn_syn_lifetime: 20
net.inet.ip.fw.dyn_ack_lifetime: 300
net.inet.ip.fw.dyn_max: 16384
net.inet.ip.fw.dyn_count: 0
net.inet.ip.fw.curr_dyn_buckets: 256
net.inet.ip.fw.dyn_buckets: 256
net.inet.ip.fw.enable: 1
net.inet.ip.fw.static_count: 25
net.inet.ip.fw.default_to_accept: 0
net.inet.ip.fw.tables_sets: 0
net.inet.ip.fw.tables_max: 128
net.inet.ip.fw.default_rule: 65535
net.inet.ip.fw.verbose_limit: 0
net.inet.ip.fw.verbose: 0
net.inet.ip.fw.autoinc_step: 100
net.inet.ip.fw.one_pass: 0

Thanks!

Michael

-- 
www.muenz-it.de
- Cisco, Linux, Networks