ipfw stateful rules and quick port re-use

[Andriy Gapon]

Recently we encountered an interesting issue at work.
By accident our software started to quickly re-use a source TCP port when
connecting to a remote system.  That is, after a graceful shutdown of a
connection (two FINs, etc), the software would quickly establish an identical
connection by re-using the same local port and connecting to the same remote
end-point.

That did not work well for the application :)
We saw problems where packets from the second connection would be dropped by
ipfw.  That happened because there would be no dynamic rule to let the packets
through even though the first connection worked without any issues.

>From a quick glance at the code it seems that the TCP protocol state kept by
ipfw for dynamic rules is "append-only".  That is, bits can be set in it but
never cleared.  So, when the first connection is closed the dynamic has "both
syn" and "both fin" bits.  When the second connection is established before the
rule is expired, the rule is re-used for it, but its state remains the same.
And its expiry time remains dyn_fin_lifetime.  I think that that opens a race
between the expiry timer (running every second) and the connection's packets
given the short lifetime.

Maybe I misanalyzed the situation and it's probably very rare.
But still it's a valid use of TCP, so maybe ipfw could support it better (e.g.,
by detecting "syn" after "both fin").

-- 
Andriy Gapon