ip_reass() fails to reassemble fragmented out-of-order traffic

[Daniel Hartmeier]

On Mon, Jul 09, 2012 at 06:13:53PM +0700, Eugene Grosbein wrote:

> tcpdump shows no errors in fragment's checksums.
> Still, they were not reassembled.

I fed your two fragments (with libdnet) to ip_input.c ip_reass() with
debug printfs added, it reassembles them fine, even when in reversed order:

Jul 10 09:06:03 testA kernel: ip_reass() called ip_id 0x87EE
Jul 10 09:06:03 testA kernel: ip_reass: no queue
Jul 10 09:06:03 testA kernel: ip_reass: IP_MF clear
Jul 10 09:06:03 testA kernel: ip_reass: created queue
Jul 10 09:06:03 testA kernel: ip_reass: returning NULL
Jul 10 09:06:03 testA kernel: ip_reass() called ip_id 0x87EE
Jul 10 09:06:03 testA kernel: ip_reass: found queue
Jul 10 09:06:03 testA kernel: ip_reass: IP_MF set
Jul 10 09:06:03 testA kernel: ip_reass: no previous segment
Jul 10 09:06:03 testA kernel: ip_reass: complete? ip_off 0, next 0
Jul 10 09:06:03 testA kernel: ip_reass: complete? ip_off 1480, next 1480
Jul 10 09:06:03 testA kernel: ip_reass: complete!
Jul 10 09:06:03 testA kernel: ip_reass: concated
Jul 10 09:06:03 testA kernel: ip_reass: m_fixhdr() 1500 -> 1501
Jul 10 09:06:03 testA kernel: ip_reass: returning non-NULL, m_len 1500, ip_len 1501
Jul 10 09:06:03 testA kernel: mbuf: 0xc4bb3c00 len: 1500, next: 0xc4fdb000, 803<pkthdr,ext>,
Jul 10 09:06:03 testA kernel: mbuf: 0xc4fdb000 len: 1, next: 0, 3<pkthdr,ext>
Jul 10 09:06:03 testA kernel: ip_input: ip_reass returned m_len 1500, ip_len 1501, hlen 20

And I see the ping with tcpdump on gif0

testA# ifconfig gif0 create
testA# ifconfig gif0 tunnel outer-me outer-peer
testA# ifconfig gif0 inet 192.168.50.10 192.168.50.9 netmask 255.255.255.0 up
testA# tcpdump -s 1600 -nvvvi gif0
tcpdump: listening on gif0, link-type NULL (BSD loopback), capture size 1600 bytes
10:12:14.028436 IP (tos 0x0, ttl 128, id 61059, offset 0, flags [DF], proto ICMP (1), length 1481)
    192.168.50.10 > 192.168.50.9: ICMP echo request, id 39570, seq 0, length 1461
^C

netstat -s diff:

-       4 fragments received
+       6 fragments received
-       2 packets reassembled ok
+       3 packets reassembled ok

        Input histogram:
-               echo: 1
-       1 message response generated
+               echo: 2
+       2 message responses generated

        Output histogram:
-               echo reply: 1
+               echo reply: 2


Are any netstat -s counters increasing that show an error (fragments
dropped, smaller than minimum)?

Do you have lots of fragments in flight at the same time, hitting the
fragment cache limit?

testA# sysctl -a | grep maxfrag
net.inet.ip.maxfragpackets: 800
net.inet.ip.maxfragsperpacket: 16

Are you running any pfil consumers (ipfilter, ipfw, pf), maybe
unintentionally (with empty ruleset)? If so, can you try disabling them?

Daniel