Deterministic lockup / panic in networking stack with ipfw / natd enabled on recent amd64 STABLE / CURRENT

[Garrett Cooper]

On Fri, Jul 2, 2010 at 8:26 PM, Ian Smith <smithi at nimnet.asn.au> wrote:
> On Sat, 3 Jul 2010, Ian Smith wrote:
>  > On Tue, 15 Jun 2010, Garrett Cooper wrote:
>  >  > Hi,
>  >  >     I'm experiencing a deterministic situation on a development box I
>  >  > manage when I do the following to enable ipfw and natd to bridge a
>  >  > network with two bce(4) enabled NICs, where if I do the following
>  >  > steps below, then try to push a few tcp frames through, the kernel
>  >  > either hardlocks, or panics in the bce(4) code, ipfw(4) code or
>  >  > networking stack code.
>  >  >     My kernel is relatively vanilla (I just turned off a number of
>  >  > drivers that I don't use because the hardware support isn't there),
>  >  > and all of the networking options available in GENERIC are enabled as
>  >  > well. I have ipfw, ipfw_nat, and libalias built as modules, along with
>  >  > bce and em.
>  >  >     I've included the stats on the machine. Note that it is a dual
>  >  > SMT-enabled quad core machine with 8GB of RAM. I haven't done anything
>  >  > to pimp the box settings via make.conf whatsoever. I would provide a
>  >  > crashdump, but dumpon is broken on the box (which is extremely
>  >  > annoying). Please note that pf doesn't have any issues pushing packets
>  >  > with similar rules.
>  >  >     This has occurred on both 8-STABLE (r209169), and 9-CURRENT (r208809).
>  >  >     Here's the manual procedure for reproducing the issue:
>  >  >
>  >  > # Do the following steps (this isn't automated apparently as it
>  >  > completely blocks off a running box, when using ipfw restart is run).
>  >  >
>  >  > # Copy the 8.0-RELEASE copy of rc.firewall over
>  >  > cp -p /usr/src/etc/rc.firewall /etc
>  >  >
>  >  > # Make sure you have access via ssh being redirected via natd.
>  >  > echo "redirect_port tcp 192.168.10.1:22 22" > /etc/natd.conf
>  >  >
>  >  > # Enable all of the required services and knobs
>  >  > cat >> /etc/rc.conf <<EOF
>  >  > firewall_enable="YES"
>  >  > firewall_logging="YES"
>  >  > firewall_nat_enable="YES"
>  >  > firewall_nat_interface="bce1"
>  >  > firewall_type="open"
>  >  > gateway_enable="YES"
>  >  > ipfw_enable="YES"
>  >  > natd_enable="YES"
>  >  > natd_interface="bce1"
>  >  > natd_flags="-dynamic -d -m"
>  >  > EOF
>  >
>  > Garrett I missed this earlier; here from your ref in the TSO thread.
>  >
>  > If you enable both firewall_nat and natd as above, on that config you
>  > should have wound up with two of ipfw rule 50, like
>  >
>  >  50 divert 8668 ip4 from any to any via bce1
>  >  50 nat 123 ip4 from any to any via bce1
>  >
>  > but I don't think you really wanted to run natd then firewall_nat again
>  > like that?
>
> Oh, sorry .. that's not right; I quite forgot the discussions in ipfw@
> about this a while ago, until I re-browsed natd(8):
>
>          After translation by natd, packets re-enter the firewall at the rule
>          number following the rule number that caused the diversion (not the
>          next rule if there are several at the same number).
>
> so in this case only natd should be invoked and the ipfw nat skipped.
>
>  > Also I'm pretty sure you'd need to include '-f /etc/natd.conf' in your
>  > natd_flags for your redirect_port config, here's no default configfile
>  > for natd (AFAIK)
>
> I think that's right - or you can specify -redirect_port in natd_flags.
>
>  > I guess rc.firewall ought to be checking that natd_enable and
>  > firewall_nat_enable aren't both YES ..
>
> .. and that becomes irrelevant, though it's still an ambiguous config.

    I'll look into this more closely on Sunday when I come in to repro
the issue.
Thanks for the feedback :)!
-Garrett