cxgbetool & hw filtering issues

[Alexander V. Chernikov]

Hello list!

While experimenting with Chelsio T440-CR (cxgbe) internal firewall, I'm 
getting some kind of unexpected results:

filtering 'type ipv4 action drop' permits IPv4 TCP traffic with bad 
checksum.
filtering 'type IPv6 action drop' permits IPv6 traffic to multicast 
addresses (MLDv2, etc..)
filtering 'ethtype 34525 action drop' (drop all IPv6) results in 
'CHELSIO_T4_SET_FILTER: Argument list too long' despite to what is said 
in budget table from cxgbetool.8
filtering 'matchtype 4 action drop' or similar (4,5,4:0,4:4, 5:0, 5:5) 
does not match anything despite some traffic definitely falls into that 
conditions.
filtering 'action drop' and 'iport X action drop' filters IPv4 traffic only.
filter 'type ipv6 ...' can be set on (0,4,8,12,...) filter numbers 
yelling 'CHELSIO_T4_SET_FILTER: Invalid argument' on other numbers.

What can I do to debug further/fix this behavior?

Some more questions:
Does anybody known how I can get/set total number of HW firewall 
records? There is such tunable in Linux version.
Is there any way to retrieve _host_ interface statistic (e.g. how much 
traffic in packets/bytes are thrown to NIC driver)?



Setup description:

[packet generator replaying small PCAP with 280kpps rate] -> cxgbe3 
[[FreeBSD 10-CURRENT r248721]].

PCAP is captured on my host machine so
1) Outgoing TCP checksums are almost all wrong
2) DST macs are not modified (so they are all unknown to NIC).

cxgbe3: 
flags=28943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,PPROMISC> 
metric 0 mtu 1500
options=6c00bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
         ether 00:07:43:11:88:d8
         inet6 fe80::207:43ff:fe11:88d8%cxgbe3 prefixlen 64 scopeid 0x9
         inet6 2a02:6b8:0:401:207:43ff:fe11:88d8 prefixlen 64 detached 
deprecated autoconf
         nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
         media: Ethernet 10Gbase-Twinax <full-duplex>
         status: active

dev.t4nex.0.%desc: Chelsio T440-CR NIC (rev 2), S/N:PT42110574, 
E/C:01234567890123
dev.t4nex.0.%driver: t4nex
dev.t4nex.0.%location: slot=0 function=4
dev.t4nex.0.%pnpinfo: vendor=0x1425 device=0x4403 subvendor=0x1425 
subdevice=0x0000 class=0x020000
dev.t4nex.0.%parent: pci8
dev.t4nex.0.nports: 4
dev.t4nex.0.hw_revision: 2
dev.t4nex.0.firmware_version: 1.8.4.0
dev.t4nex.0.cf: default
dev.t4nex.0.cfcsum: 4260083439
dev.t4nex.0.linkcaps: 0
dev.t4nex.0.niccaps: 1<NIC>
dev.t4nex.0.toecaps: 0
dev.t4nex.0.rdmacaps: 0
dev.t4nex.0.iscsicaps: 0
dev.t4nex.0.fcoecaps: 0
dev.t4nex.0.core_clock: 228125
dev.t4nex.0.holdoff_timers: 1 5 10 50 100 200
dev.t4nex.0.holdoff_pkt_counts: 1 8 16 32
dev.t4nex.0.fwq.abs_id: 0
dev.t4nex.0.fwq.cntxt_id: 0
dev.t4nex.0.fwq.cidx: 121
dev.t4nex.0.mgmtq.cntxt_id: 0
dev.t4nex.0.mgmtq.cidx: 95
dev.t4nex.0.mgmtq.pidx: 111
dev.t4nex.0.mgmtq.tx_wrs: 119
dev.t4nex.0.mgmtq.no_desc: 0
dev.t4nex.0.mgmtq.unstalled: 0

# kenv | grep cxgbe
hw.cxgbe.fcoecaps_allowed="0"
hw.cxgbe.iscsicaps_allowed="0"
hw.cxgbe.nrxq10g="4"
hw.cxgbe.ntxq10g="4"
hw.cxgbe.qsize_rxq="4096"
hw.cxgbe.qsize_txq="4096"
hw.cxgbe.rdmacaps_allowed="0"
hw.cxgbe.toecaps_allowed="0"


TRAFFIC PART:
             input       (cxgbe3)           output
    packets  errs idrops      bytes    packets  errs      bytes colls
     284368     0     0   85436494          0     0          0     0
     284340     0     0   85442168          0     0          0     0
     284205     0     0   85464055          0     0          0     0
...
(not changing, nearly constant rate, is not affected by filters)

# ipfw show 200
00200      16860      2685762 deny ip from any to any via cxgbe3

# Running counter to see how much is actually dropped/passed
# while true; do sleep 1; ipfw show 200 ; ipfw -q zero 200 ;done
[[ empty filters ]]
00200     281769     80351685 deny ip from any to any via cxgbe3
..
[[ ### (1) IPv4 EXPERIMENT ]]
[[ # ./cxgbetool t4nex0 filter 0 type ipv4 action drop ]]
00200     115263     15431259 deny ip from any to any via cxgbe3
00200     116523     15584332 deny ip from any to any via cxgbe3

[[# time tcpdump -i cxgbe3 -lnps0 -c 100 ip
18:18:42.621728 IP 95.108.170.36.39215 > 93.158.158.93.80: Flags [.], 
ack 4252241156, win 995, options [nop,nop,TS val 538195932 ecr 
1194270183], length 0
..
tcpdump -i cxgbe3 -lnps0 -c 100 ip  0,00s user 0,01s system 15% cpu 
0,059 total
#]]

[[ ### (2) IPv6 EXPERIMENT ]]
[[ # ./cxgbetool t4nex0 filter 4 type ipv6 action drop ]]
00200      64962     10332022 deny ip from any to any via cxgbe3
00200      64878     10327694 deny ip from any to any via cxgbe3
...
[[# time tcpdump -i cxgbe3 -lnps0 -c 100 ip6
18:21:34.553596 IP6 fe80::884:a1e8:86ae:57f7 > ff02::16: HBH ICMP6, 
multicast listener report v2, 3 group record(s), length 68
..
tcpdump -i cxgbe3 -lnps0 -c 100 ip6  0,00s user 0,00s system 0% cpu 
0,483 total
#]]


Address in (1) is my host machine address, viewing resulting .pcap file 
in wireshark shows incorrect TCP checksums for IPv4 packets.
Other pcaps not containing "bad" traffic are properly filtered by rules 
above.