64K NFS I/O generates a 34mbuf list for TCP which breaks TSO

[John-Mark Gurney]

Rick Macklem wrote this message on Thu, Jan 30, 2014 at 22:32 -0500:
> Adrian Chadd wrote:
> > On 30 January 2014 07:06, Rick Macklem <rmacklem at uoguelph.ca> wrote:
> > > Hi, just adding one more idea on what to do about this
> > > to the list:
> > > - Add a if_hw_tsomaxseg and modify the loop in tcp_output()
> > >   so that it uses both if_hw_tsomax and if_hw_tsomaxseg to
> > >   decide how much to hand to the device driver in each mbuf list.
> > >   (I haven't looked to see how easy it would be to change this
> > >   loop.)
> > 
> > I don't think that's a hack. I think adding that and setting
> > tsomaxseg
> > to say 30 for now would be a good comprimise.
> > 
> Well, my TCP is very rusty and I have no way to test it (I don't
> have anything that does TSO), but I've attached a stab at a patch
> to do this.
> 
> Maybe it can be used as a starting point for this, if others think
> it makes sense.
> 
> The "#ifdef notyet" in the patch would become something like:
> # if __FreeBSD_version >= NNNN
> when a change to add if_hw_tsomaxseg is done, was what I was
> thinking.

Definately need to make sure you fix the drivers that support large
enough sg arrays like ixgb which supports 100...

Just a sampling of ones that use a _SCATTER define:
./e1000/if_igb.h:#define IGB_MAX_SCATTER                64
./e1000/if_lem.h:#define EM_MAX_SCATTER         64
./e1000/if_em.h:#define EM_MAX_SCATTER          32
./nfe/if_nfereg.h:#define       NFE_MAX_SCATTER         32
./ixgbe/ixgbe.h:#define IXGBE_82598_SCATTER             100
./ixgbe/ixgbe.h:#define IXGBE_82599_SCATTER             32
./ixgb/if_ixgb.h:#define IXGB_MAX_SCATTER           100

I wonder how many of these are hardware limits, or just I don't
want to allocate too much space on the stack, as 16 bytes per
bus_dma_segment_t (on amd64) adds up...

The other question is should the drivers w/ a limit on the segments
reduce the size of the TSO packet so that we don't need to
m_defrag/m_collapse which are expensive operations...

-- 
  John-Mark Gurney				Voice: +1 415 225 5579

     "All that I will do, has been done, All that I have, has not."