Broken IPsec + enc +pf/ipfw

[Kyle Williams]

Hello,

I'm currently using 10.0, IPSEC, racoon, enc, and pf between two remote
hosts without NATT. The gif tunnel is ipv4 only, host A is ipv4 only,
host B is ipv4/ipv6. I use IPSEC to route traffic between jails on both
hosts, with the jails using cloned lo1 and 10.0.0.0/8 addresses.

I'm testing the posted patches on host A with the following pf.conf:
  block all
  pass all

I'm using the recommended sysctl's:
  net.enc.in.ipsec_bpf_mask=2
  net.enc.in.ipsec_filter_mask=2
  net.enc.out.ipsec_bpf_mask=1
  net.enc.out.ipsec_filter_mask=1

On Tue Oct 21 00:50:33 2014, Andrey V. Elsukov wrote:
> On 20.10.2014 20:18, Matthew Grooms wrote:
>> Lastly, I tried to locate a relevant PR but didn't find anything
>> concrete. Is this related to the issue? And if so, can it be MFCd?
>>
>> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=110959
>
> Did you try the patch from last PR? It is small and should be applicable
> to stable/10.

On 10/20/2014 3:50 PM, Andrey V. Elsukov wrote:
>This commit fixes similar problem with ipfw in 11.0-CURRENT. But I think
>it won't help you with pf in 10. I guess r266800 is what you need.

On Tue Oct 21 11:57:44 2014, Andrey V. Elsukov wrote:
>It also changes places from where pfil consumers are called. You may use
>dtrace script to see where is the problem. Try this:
>
>> kldload dtraceall
>> cat > ~/ipsec.d
>#!/usr/sbin/dtrace -s
>
>fbt::ipsec_filter:entry
>{
>        m = *(struct mbuf **)arg0;
>        ip = (struct ip *)m->m_hdr.mh_data;
>        printf("%s: %s: %s->%s proto %d",
>            (arg1 & 1) ? "in": "out",
>            (arg2 & 1) ? "before": "after",
>            inet_ntoa(&ip->ip_src.s_addr),
>            inet_ntoa(&ip->ip_dst.s_addr),
>            ip->ip_p);
>}
>^D
>> chmod +x ~/ipsec.d
>> ~/ipsec.d
>
>This script will print messages when ipsec_filter function will be
>invoked. Can you show what it will print for your case?

IPSEC traffic is still blocked after applying r272695. Here is the output
of the above script during an incoming wget:
  2  26761               ipsec_filter:entry out: before: 10.0.5.10->10.0.4.2 proto 6
  2  26761               ipsec_filter:entry out: before: 10.0.5.10->10.0.4.2 proto 6
  2  26761               ipsec_filter:entry out: before: 10.0.5.10->10.0.4.2 proto 6
  ...

With pf.conf only containing `pass all', traffic is passed (IP's munged):
  2  26761               ipsec_filter:entry out: before: 10.0.5.10->10.0.4.2 proto 6
  2  26761               ipsec_filter:entry out: after: 1.1.1.1->2.2.2.2 proto 4
  1  26761               ipsec_filter:entry out: before: 10.0.5.10->10.0.4.2 proto 6
  1  26761               ipsec_filter:entry out: after: 1.1.1.1->2.2.2.2 proto 4
  1  26761               ipsec_filter:entry out: before: 10.0.5.10->10.0.4.2 proto 6
  1  26761               ipsec_filter:entry out: after: 1.1.1.1->2.2.2.2 proto 4
  2  26761               ipsec_filter:entry out: before: 10.0.5.10->10.0.4.2 proto 6
  2  26761               ipsec_filter:entry out: after: 1.1.1.1->2.2.2.2 proto 4
  1  26761               ipsec_filter:entry out: before: 10.0.5.10->10.0.4.2 proto 6
  1  26761               ipsec_filter:entry out: after: 1.1.1.1->2.2.2.2 proto 4
  ...

Applying r266800 failed on sys/netipsec/ipsec_output.c on the last chunk,
so I copied ipsec_output.c from head. r272695 is included as well.
Traffic is still blocked with `block all' and `pass all' in pf.conf. Here
is the dtrace output:
  0  26759               ipsec_filter:entry in: after: 10.0.4.2->10.0.5.10 proto 6
  3  26759               ipsec_filter:entry out: before: 10.0.5.10->10.0.4.2 proto 6
  0  26759               ipsec_filter:entry in: after: 10.0.4.2->10.0.5.10 proto 6
  3  26759               ipsec_filter:entry out: before: 10.0.5.10->10.0.4.2 proto 6
  0  26759               ipsec_filter:entry in: after: 10.0.4.2->10.0.5.10 proto 6
  3  26759               ipsec_filter:entry out: before: 10.0.5.10->10.0.4.2 proto 6
  0  26759               ipsec_filter:entry in: after: 10.0.4.2->10.0.5.10 proto 6
  ...

And finally with only `pass all', traffic is passed:
  0  26759               ipsec_filter:entry in: after: 10.0.4.2->10.0.5.10 proto 6
  3  26759               ipsec_filter:entry out: before: 10.0.5.10->10.0.4.2 proto 6
  3  26759               ipsec_filter:entry out: after: 1.1.1.1->2.2.2.2 proto 4
  0  26759               ipsec_filter:entry in: after: 10.0.4.2->10.0.5.10 proto 6
  3  26759               ipsec_filter:entry in: after: 10.0.4.2->10.0.5.10 proto 6
  3  26759               ipsec_filter:entry out: before: 10.0.5.10->10.0.4.2 proto 6
  3  26759               ipsec_filter:entry out: after: 1.1.1.1->2.2.2.2 proto 4
  0  26759               ipsec_filter:entry in: after: 10.0.4.2->10.0.5.10 proto 6
  0  26759               ipsec_filter:entry out: before: 10.0.5.10->10.0.4.2 proto 6
  0  26759               ipsec_filter:entry out: after: 1.1.1.1->2.2.2.2 proto 4
  ...

I'm willing to test more kernel patches, but I can't install head.

-- 
Kyle Williams
(541) 250 0314
Kyle at 1970Jan1-epo.ch
PGP key: 0xD1E5BADF