Issues with em(4) device under FreeBSD 8.0

[Pyun YongHyeon]

On Thu, Feb 18, 2010 at 07:32:48AM -0800, Patrick Mahan wrote:
> All,
> 
> I have seen a few mentions on the mailing lists in regard to issues
> with em(4) and FreeBSD 8.0 with regard to throughput.
> 
> We are also seeing similar issues on HP Proliant systems with
> this HP GE interfaces.  Previously we were running FreeBSD 6.2 and
> iperf was showing ~900 Mbits/sec between two directly connected
> systems.  After the upgrade, iperf only shows around ~350 Mbits/sec.
> 
> This seems only to be happening on the HP's.  When we upgrade another
> x86 box (privately built) we are seeing ~900 Mbits/sec even to
> one of the HP systems.
> 
> I haven't seen anything yet to account for this behavior.  Has anyone
> else seen similar issues?
> 

I know there is a possible Tx checksum offloading issue but testing
with iperf may not hit the case. One of the big change made in 8.x
is switching to buf_ring which will take advantage of multi TX
queues. So the only guess I have is TCP segment reordering caused
by em(4) and buf_ring interface. Can you capture the traffic on
receiver side and check whether you see out-of-ordered TCP segment
delivery? If the theory is right you may see the lots of
out-of-ordered segments(this could be easily checked on receiver
side with netstat -s after clearing the stats) and it will be more
frequently seen when TCP window scaling is used.
Personally I had trouble to reproduce it on my environments. It may
depend on specific workloads.

> Thanks,
> 
> Patrick