em driver input errors

[Barney Cordoba]

--- On Fri, 9/4/09, Manish Vachharajani <manishv at lineratesystems.com> wrote:

> From: Manish Vachharajani <manishv at lineratesystems.com>
> Subject: Re: em driver input errors
> To: alexpalias-bsdnet at yahoo.com
> Cc: freebsd-net at freebsd.org, "Artis Caune" <artis.caune at gmail.com>
> Date: Friday, September 4, 2009, 1:41 PM
> Just decided to follow this thread as
> it seems to be related to some
> issues we are seeing as well.
> 
> It appears that under heavy packet loads, the kernel cannot
> pull
> packets off the NIC fast enough and thus is slow to free
> up
> descriptors into which the NIC can DMA packets.  This
> causes the NIC
> to drop the packet after it's internal queue fills up (and
> record the
> packet as missed) because the hardware does not have
> enough
> descriptors to write the packets into.  We ahve this
> issue with the
> ixgbe 10 Gb/s card though the absolute packet rates at
> which we see a
> problem are higher than those reported here.
> 
> In our test scenario the problem gets worse with many
> simultaneous TCP
> connections, but the issue is the same.  Under high
> packet rates, the
> driver cannot keep up and the NIC reports missed
> packets.  The issue
> is not related to data throughput though as turning on
> jumbo frames
> solves our issue for a fixed number of connections, and it
> seems here
> that reducing the packet rate makes the misses go
> away.  More
> importantly, in our tests, only the receiver sees a
> problem, the
> transmitter is fine.
> 
> There was also another thread about problems with UDP
> throughput that
> I suspect are caused by the same type of packet rate
> spikes.
> 
> The question is, why is the kernel stack slow to handle
> these packet
> rates, doing some back of the envelope calculations, they
> don't seem
> too bad?  Where is the time going?  And, are our
> problem, the UDP
> issue, and this problem all caused by the same source of
> slowness or
> are they three unrelated issues.
> 
> Manish

What specific kinds of input errors are you getting?

How many PPS are you doing, what is the size of the ring, and
the interrupt modulation rate?

Are the NICs PCIe or PCIx?

Barney