igb(4) Raising IGB_MAX_TXD ??

[Sean Bruno]

ok, good.  that at least confirms that I correctly translated between
the driver code and documented specification.  

I will try 8k as a test for now and see how that runs.  

sean

On Wed, 2012-04-18 at 09:46 -0700, Jack Vogel wrote:
> The MAX value is something I set, not a hardware thing, it was based
> on reports
> I had from the various driver engineers in our org. If you increase
> the ring size 
> you might run into other performance issues, however there's nothing
> stopping
> you from trying. Just be aware that its not something that's been
> tested.
> 
> Let me know how it goes please :)
> 
> Jack
> 
> 
> On Wed, Apr 18, 2012 at 9:27 AM, Sean Bruno <seanbru at yahoo-inc.com>
> wrote:
>         On Wed, 2012-04-18 at 00:28 -0700, Luigi Rizzo wrote:
>         > On Tue, Apr 17, 2012 at 04:24:24PM -0700, Sean Bruno wrote:
>         > > We're running a service with a 82576 configured with 4
>         queues and a
>         > > maxed rxd/txd configuration:
>         > >
>         > > http://people.freebsd.org/~sbruno/igb_stats.txt
>         >
>         > these stats show that over half of your incoming traffic is
>         > made of small packets (65..127 bytes) but especially, that
>         > the "missed packets" count is very small (18k out of 40G
>         packets)
>         > none of them is reported as "no_desc_avail", and only 76 are
>         > "recv_no_buffer".
>         >
>         > Are you dropping packets in the ip interrupt handler by
>         chance ?
>         > what are your settings there ?
>         >
>         
>         nope, doesn't look like it.
>         
>         http://people.freebsd.org/~sbruno/igb_ip_stats.txt
>         
>         > BTW it seems that there is only one global setting for the
>         dispatch
>         > policy, but for instance there are two netisr_dispatch()
>         calls
>         > in the incoming path, one for layer2 and one for layer3.
>         > The former has relatively little work to do and so it might
>         > make sense to have direct dispatch, the other can be
>         expensive
>         > so i wonder if it wouldn't be better to use deferred
>         dispatch.
>         > If not, perhaps you might try to reduce the
>         rx_processing_limit
>         > to bring down the load on the intr thread.
>         
>         
>         I don't really see any issue with horsepower on this host at
>         the moment
>         with 4 queues.  I mean it looks a little something like this
>         under high
>         load:
>         
>         http://people.freebsd.org/~sbruno/igb_top.txt
>         
>         I guess my question still stands though, since the ethernet
>         controller
>         is reporting that it doesn't have any more descriptors
>         available is the
>         hardcoded 4k max descriptors a limit that an be raised?
>         
>         > With your numbers i doubt that raising the queue size helps.
>         
>         
>         Indeed, you're probably right and this is more than likely an
>         application problem that will have to be resolved.  However,
>         I'm still
>         curious if the MAX_RXD/TXD is really 4k or if the
>         documentation is
>         correct and we can raise it to 32k for testing?
>         
>         Sean
>