Efficient use of Dummynet pipes in IPFW

[Brett Glass]

For years, we've used "Dummynet" in FreeBSD for bandwidth control. 
Unfortunately, the semantics of IPFW can, at times, make the use of 
Dummynet awkward and inefficient. For example, suppose you want to 
create a set of rules that does bandwidth limiting first
and then blocks certain ports (e.g. TCP ports 137 through 139). You 
want to throttle first and then block ports, so that (a) blocked 
packets count against the user's bandwidth limit and (b) a flood of 
packets will be bandwidth-limited before it runs
through the rest of the rules.

If net.ip.fw.one_pass is set to 0, packets emerging from a Dummynet 
pipe or queue will re-emerge at the next rule. This is good, 
because the packet can be passed on to the rules that block ports. 
But there's a problem: you usually do not want to go to the next 
rule (which is likely to be one that tests the packet to see if it 
is to go into a different Dummynet pipe). Rather, you want the 
packet to next be tested against a rule farther down -- after all 
of the rules involving bandwidth limiting.

Here's an example of what I mean. Suppose you have several groups 
of users, at IP addresses 0.0.0.1, 0.0.0.2, etc. Each group has a 
separate pipe regulating its bandwidth consumption. You might have 
rules like this:

# First group

${fwcmd} pipe 1 config bw 512kbit/s
${fwcmd} pipe 2 config bw 512kbit/s

${fwcmd} add pipe 1 ip from 0.0.0.0/24{55,56,57} to any in via fxp1
${fwcmd} add pipe 2 ip from any to 0.0.0.0/24{55,56,57} out via fxp1

# Second group

${fwcmd} pipe 3 config bw 1024bit/s
${fwcmd} pipe 4 config bw 1024kbit/s

${fwcmd} add pipe 3 ip from 0.0.0.0/24{35-40} to any in via fxp1
${fwcmd} add pipe 4 ip from any to 0.0.0.0/24{35-40} out via fxp1

# Filtering here

What you'd really like is to have any packet that satisfies one of 
the "pipe" rules jump down to the filtering rules after being 
reinjected into IPFW.

Unfortunately, because IPFW doesn't have a "not" that can cover the 
"and" of all the conditions in the rule -- that is, you can't say 
"not (ip from A to any in via fxp1)" -- it's very difficult to do 
this with a single rule containing a "skipto" action. What's more, 
there's no "resume at" clause available in IPFW that would change 
where a packet was reinjected, and no such thing as a "come from" 
directive (something that's often joked about in programming 
classes). So, what's the best way get a packet to skip past the 
remaining bandwidth limiting rules once it was selected to go into a pipe?

--Brett Glass