5.4 amd64 kernel and em driver issue ...

[Matthew Grooms]

All,

      Has anyone done any extensive testing with the em driver on a 5.4 
release amd64 SMP kernel? I have two boxes in a firewall setup that 
contain 6 em interfaces each. The public interface on both of them ( em0 
) will simply stop transmitting and then start working again after some 
time all by themselves.

      I did a lot of testing with 5.3 release candidates and did not see 
this behavior. Did anything go into the 5.4 kernel late in the release 
cycle that could have effected this?

My kernel config is GENERIC with the following modifications ...

1) removal of IPV6 and faith
2) removal of USB, USB Ethernet and Firewire
3) addition of SMP support
4) addition of pf, pflog, pfsync, carp and ALTQ

Detailed description of the problem ...

 From the firewall itself I could be pinging www.google.com and it will 
just stop. After a few minutes to an hour or so later it will just start 
working again. The really odd thing is that I can always ssh into the 
box on private em interface. The really really odd thing is that I can 
run tcpdump the public interface ( that I can't talk out of ) while the 
problem occurs and see traffic on the wire like ...

1) ICMP packets still coming from ping on my firewall to google
    ( maybe BPF picks it up early and the interface is dropping it ? )
2) ARP requests
3) CDP advertisements
4) Misc other broadcast traffic

What I have tried so far to diagnose the issue ...

1) disabling pf using -d
2) disabling SMP in kernel
3) disabling carp in kernel
4) disabling ALTQ in kernel
5) hard coding the link speed to either half or full duplex
6) trimming down my route table
7) replacing both network cables
8) moving to different ports on the switch
9) moving to a different switch all together
10) running with mpsafenet disabled

What I am testing right now ...

1) disabling pf in the kernel
2) disabling HTT in hardware
3) disabling USB & Firewire in hardware
4) sacrificing a chicken on the alter of the Ethernet gods

Any help is _GREATLY_ appreciated as I have to get these boxes out into 
production quickly. Am I missing something obvious? Could I be having a 
resource conflict somehow? Could I be missing a lock assertion or LOR 
for lack of witness or invariants? I will do whatever I can to provide 
any info to help diagnose this problem. For starters, here is my kernel 
config and dmesg output.

http://hole.shrew.net/~mgrooms/files/freebsd/custom.txt
http://hole.shrew.net/~mgrooms/files/freebsd/dmesg.txt

Thanks in advance,

-Matthew