Traffic mysteriously dropping

[Christopher McGee]

I have 2 firewalls using all "em" network cards.  They have 2 onboard 
Intel Gigabit interfaces and 1 quad port intel pro1000MT in each 
firewall.  They are currently using both of the onboard interfaces and 2 
of the interfaces from the pci cards.  The firewalls are running carp 
and pfsync for failover.  They are managing traffic for a gigabit link 
and they usually don't push more than 150-200 Mbit/s and that is rare.  
Some http traffic is mysteriously just disappearing, even at times when 
the firewalls are not busy(only 3-4 Mbit/s of traffic).  I've tested 
this, and the traffic is reaching the firewall(inbound to our network) 
and hits pf and seems to be passing but then just never makes it out the 
other interfaces(although pf does not log any blocked packets).  The 
client will resend SYN packets until the connection eventually just 
times out.  This timeout is happening on approximately 1 out of 25 
connections. 

Here is how I fixed this temporarily:
I moved the rule for the http traffic to the FIRST rule of pf.conf and 
make it a quick rule and bidirectional(stateless), it works and doesn't 
seem to drop any connections.

I have a fairly extensive ruleset, 378 rules to be exact when they are 
all loaded.  I am using if-bound states.  If I make these rules 
stateful, or move them down even one or 2 lines in the list of rules, 
they start dropping connections again.  Hopefully someone can help with 
this.

Chris