igb(4) Raising IGB_MAX_TXD ??

[Sean Bruno]

On Wed, 2012-04-18 at 00:28 -0700, Luigi Rizzo wrote:
> On Tue, Apr 17, 2012 at 04:24:24PM -0700, Sean Bruno wrote:
> > We're running a service with a 82576 configured with 4 queues and a
> > maxed rxd/txd configuration:
> > 
> > http://people.freebsd.org/~sbruno/igb_stats.txt
> 
> these stats show that over half of your incoming traffic is
> made of small packets (65..127 bytes) but especially, that
> the "missed packets" count is very small (18k out of 40G packets)
> none of them is reported as "no_desc_avail", and only 76 are
> "recv_no_buffer".
> 
> Are you dropping packets in the ip interrupt handler by chance ?
> what are your settings there ?
> 
nope, doesn't look like it.  

http://people.freebsd.org/~sbruno/igb_ip_stats.txt

> BTW it seems that there is only one global setting for the dispatch
> policy, but for instance there are two netisr_dispatch() calls
> in the incoming path, one for layer2 and one for layer3.
> The former has relatively little work to do and so it might
> make sense to have direct dispatch, the other can be expensive
> so i wonder if it wouldn't be better to use deferred dispatch.
> If not, perhaps you might try to reduce the rx_processing_limit
> to bring down the load on the intr thread.

I don't really see any issue with horsepower on this host at the moment
with 4 queues.  I mean it looks a little something like this under high
load:

http://people.freebsd.org/~sbruno/igb_top.txt

I guess my question still stands though, since the ethernet controller
is reporting that it doesn't have any more descriptors available is the
hardcoded 4k max descriptors a limit that an be raised?

> With your numbers i doubt that raising the queue size helps.

Indeed, you're probably right and this is more than likely an
application problem that will have to be resolved.  However, I'm still
curious if the MAX_RXD/TXD is really 4k or if the documentation is
correct and we can raise it to 32k for testing?

Sean
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 488 bytes
Desc: This is a digitally signed message part
Url : http://lists.freebsd.org/pipermail/freebsd-net/attachments/20120418/d8ffb02a/attachment.pgp