IPFW with NAT : Problems with duplicate packets on FreeBSD 10.3-RC3

Thu Apr 7 23:11:28 UTC 2016

> On 7 Apr 2016, at 17:08, Dr Josef Karthauser <joe at truespeed.com> wrote:
> 
> Looks like the first packet is being retransmitted, which means that the nat is probably misconfigured and the TCP connection is broken in some strange way.
> 
> Does anyone have a clue as to where to look? The ipfw rules are simple enough - what have I missed?

Ok, the packet definitely isn’t being retransmitted. I’ve done a tcpdump/pcap capture and taken a look and I get a packet that I’ve included below.

It’s got a 'HTTP/1.1 200 OK’ inserted mid-flow right in the middle of an HTTP response. Looking at this I’d be inclined to think it’s a bug in the webserver/tomcat, however, what’s strange is that if I ‘curl' the jailed web server directly from the host machine on the private IP address (bypassing the NAT), the HTTP response  received is perfectly fine. It’s only when I do an HTTP request to the public IP address and go through the NAT that I experience the problem.

How could this happen? Is it a buggy packet reassembly in the kernel perhaps?

Joe

p.s here’s the strange packet with an HTTP response injected in the middle of a HTML stream:

23:01:07.204016 IP (tos 0x0, ttl 64, id 4190, offset 0, flags [DF], proto TCP (6), length 1500)
    31.210.26.216.8080 > infiniverse.karthauser.co.uk.62475: Flags [.], cksum 0xda1c (incorrect -> 0x7ff7), seq 8689:10137, ack 86, win 1040, options [nop,nop,TS val 124159447 ecr 1737359970], length 1448
.........g.).............
.f..g..b       <h4>Other Documentation</h4>
                        <ul>
                            <li><a href="http://tomcat.apache.org/connectors-doc/">Tomcat Connectors</a></li>
                            <li><a href="http://tomcat.apache.org/connectors-doc/">mod_jk Documentation</a></li>
                        HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Content-Type: text/html;charset=ISO-8859-1
Transfer-Encoding: chunked
Date: Thu, 07 Apr 2016 23:01:05 GMT

2000

<!DOCTYPE html>

<html lang="en">
    <head>
        <title>Apache Tomcat/7.0.68</title>
        <link href="favicon.ico" rel="icon" type="image/x-icon" />
        <link href="favicon.ico" rel="shortcut icon" type="image/x-icon" />
        <link href="tomcat.css" rel="stylesheet" type="text/css" />
    </head>

    <body>
        <div id="wrapper">
            <div id="navigation" class="curved container">
                <span id="nav-home"><a href="http://tomcat.apache.org/">Home</a></span>
                <span id="nav-hosts"><a href="/docs/">Documentation</a></span>
                <span id="nav-config"><a href="/docs/config/">Configuration</a></span>
                <span id="nav-examples"><a href="/examples/">Examples</a></span>
                <span id="nav-wiki"><a href="http://wiki.apache.org/tomcat/FrontPage">Wiki</a></span>
                <span id="nav-lists"><a href="http://tomcat.apache.org/lists.html">Mailing Lists</a></span>
                <s