Re: Throughput extremely decreases when IPFW 7000 mac based rules activated

Trimming CC: list (I'm not in ipfw@)

10.08.2021 19:14, alfadev wrote:

>> ipfw add 10 skipto 1000 not layer2 # do not MAC-filter layer3 packets that have L2 header stripped already anyway
>> ipfw add 20 allow out # do not MAC-filter packets leaving the firewall, MAC-filter incoming only
>> Also, your 11.2 version is quite old and you may need to upgrade to 11.4-STABLE at least to catch up with bugfixes and/or optimizations.
>> Also, if you do filtering bridge, you should carefully read if_bridge(4) manual page,
> 
> * I have added skipto rule for layer3
> * I have to add both in and out allow pipe rules for each MAC Address to assign bandwidth per MAC

Why do you assign bandwidth per MAC instead of per IP or (set of) subnet(s)?
ipfw is designed to process multiple sets of IP networks but not MACs.

> * I also tried this configuration on FreeBSD 12.2 but no luck same problem occurs.

FreeBSD 12 is distinct major branch and may have its own performance problems,
I did not propose using latest version. I proposed to try 11.4-STABLE that is most tested
of supported versions to the moment.

> * I have no bridge configuration
> 
>> that ipfw is not right tool for this task at the moment.
> * How can i overcome this problem without using IPFW?

Cannot tell for sure. You need something with hashed or tree-based lookup, not linear search.

Some comments on your ruleset follow.

> Thanks for any help ..
> 
> Here is my configuration:
> #################################################
> 
> ipfw -q -f flush
> 
> ipfw pipe 2 config bw 500000Kbit mask dst-ip 0xffffffff
> ipfw pipe 1002 config bw 500000Kbit mask src-ip 0xffffffff
> 
> ipfw pipe 4 config bw 1024Kbit mask dst-ip 0xffffffff
> ipfw pipe 1004 config bw 1024Kbit mask src-ip 0xffffffff
> 
> 
> # Loopback allow
> ipfw -q add 1 allow all from any to any out via lo0
> ipfw -q add 2 allow all from any to any in via lo0

You need not two different rules for loopback. Singe rule will work with less overhead:
allow all from any to any via lo0

> 
> # WAN Allow
> ipfw -q add 3 allow ip from any to any MAC any any via em0
> ipfw -q add 4 allow ip from any to any via em0

Same here, single rule is sufficient:
allow ip from any to any via em0

> # Layer2 em1 enable arp traffic
> ipfw -q add 5 allow ip from any to any layer2 mac-type arp via em1
> ipfw -q add 6 skipto 64000 all from any to any not layer2
> 
> # Layer2 blocked mac
> ipfw -q add 1189 deny ip from any to any MAC 1c:cc:d6:42:5e:xx any via em1
> ipfw -q add 2189 deny ip from any to any MAC any 1c:cc:d6:42:5e:xx via em1

This is inefficient as each incoming and outgoing frame will be matched against both rules
but only one match should be performed for incoming and one for outgoing frame.

You should first check only incoming frames:

add 1100 skipto (NNN+1) out		# filter outgoing frames later
add 1110 skipto 12000 in not recv em1	# do not filter frames coming from other interfaces here
add 1189 deny MAC any 1c:cc:d6:42:5e:xx	# less attributes, faster matching (do not match for em1 again and again)
add 1190 deny MAC any 3c:dc:bc:ab:56:yy
...
add NNN skipto 12000 # end of MAC-filter for incoming frames

So outgoing frames will not pass over whole list of rules for "MAC xxx any" they never match anyway.
Then same for another direction:

add (NNN+1) skipto 12000 out not xmit em1	# do not filter frames transmitted over other interfaces here
add (NNN+2) deny MAC 1c:cc:d6:42:5e:xx any 	# again, optimize out match for interface name, check MAC only
...

You should be VERY careful writing rules for expensive filtering.