mps driver chain_alloc_fail / performance ?

[Desai, Kashyap]

Which driver version is this ? In our 09.00.00.00 Driver (which is in pipeline to be committed) has 2048 chain buffer counter.
And our Test team has verified it with almost 150+ Drives.

As suggested by Ken, Can you try increasing MPS_CHAIN_FRAMES  to 4096 OR 2048

~ Kashyap

> -----Original Message-----
> From: owner-freebsd-scsi at freebsd.org [mailto:owner-freebsd-
> scsi at freebsd.org] On Behalf Of Kenneth D. Merry
> Sent: Sunday, January 15, 2012 4:53 AM
> To: John
> Cc: freebsd-scsi at freebsd.org
> Subject: Re: mps driver chain_alloc_fail / performance ?
> 
> On Sat, Jan 14, 2012 at 05:16:18 +0000, John wrote:
> > Hi Folks,
> >
> >    I've started poking through the source for this, but thought I'd
> > go ahead and post to ask other's their opinion.
> >
> >    I have a system with 3 LSI SAS hba cards installed:
> >
> > mps0: <LSI SAS2116> port 0x5000-0x50ff mem 0xf5ff0000-
> 0xf5ff3fff,0xf5f80000-0xf5fbffff irq 30 at device 0.0 on pci13
> > mps0: Firmware: 05.00.13.00
> > mps0: IOCCapabilities:
> 285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay>
> > mps1: <LSI SAS2116> port 0x7000-0x70ff mem 0xfbef0000-
> 0xfbef3fff,0xfbe80000-0xfbebffff irq 48 at device 0.0 on pci33
> > mps1: Firmware: 07.00.00.00
> > mps1: IOCCapabilities:
> 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDis
> c>
> > mps2: <LSI SAS2116> port 0x6000-0x60ff mem 0xfbcf0000-
> 0xfbcf3fff,0xfbc80000-0xfbcbffff irq 56 at device 0.0 on pci27
> > mps2: Firmware: 07.00.00.00
> > mps2: IOCCapabilities:
> 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDis
> c>
> 
> The firmware on those boards is a little old.  You might consider
> upgrading.
> 
> >    Basically, one for internal and two for external drives, for a
> total
> > of about 200 drives, ie:
> >
> > # camcontrol inquiry da10
> > pass21: <HP EG0600FBLSH HPD2> Fixed Direct Access SCSI-5 device
> > pass21: Serial Number 6XR14KYV0000B148LDKM
> > pass21: 600.000MB/s transfers, Command Queueing Enabled
> 
> That's a lot of drives!  I've only run up to 60 drives.
> 
> >    When running the system under load, I see the following reported:
> >
> > hw.mps.0.allow_multiple_tm_cmds: 0
> > hw.mps.0.io_cmds_active: 0
> > hw.mps.0.io_cmds_highwater: 772
> > hw.mps.0.chain_free: 2048
> > hw.mps.0.chain_free_lowwater: 1832
> > hw.mps.0.chain_alloc_fail: 0         <--- Ok
> >
> > hw.mps.1.allow_multiple_tm_cmds: 0
> > hw.mps.1.io_cmds_active: 0
> > hw.mps.1.io_cmds_highwater: 1019
> > hw.mps.1.chain_free: 2048
> > hw.mps.1.chain_free_lowwater: 0
> > hw.mps.1.chain_alloc_fail: 14369     <---- ??
> >
> > hw.mps.2.allow_multiple_tm_cmds: 0
> > hw.mps.2.io_cmds_active: 0
> > hw.mps.2.io_cmds_highwater: 1019
> > hw.mps.2.chain_free: 2048
> > hw.mps.2.chain_free_lowwater: 0
> > hw.mps.2.chain_alloc_fail: 13307     <---- ??
> >
> >    So finally my question (sorry, I'm long winded): What is the
> > correct way to increase the number of elements in sc->chain_list
> > so mps_alloc_chain() won't run out?
> 
> Bump MPS_CHAIN_FRAMES to something larger.  You can try 4096 and see
> what
> happens.
> 
> > static __inline struct mps_chain *
> > mps_alloc_chain(struct mps_softc *sc)
> > {
> >         struct mps_chain *chain;
> >
> >         if ((chain = TAILQ_FIRST(&sc->chain_list)) != NULL) {
> >                 TAILQ_REMOVE(&sc->chain_list, chain, chain_link);
> >                 sc->chain_free--;
> >                 if (sc->chain_free < sc->chain_free_lowwater)
> >                         sc->chain_free_lowwater = sc->chain_free;
> >         } else
> >                 sc->chain_alloc_fail++;
> >         return (chain);
> > }
> >
> >    A few layers up, it seems like it would be nice if the buffer
> > exhaustion was reported outside of debug being enabled... at least
> > maybe the first time.
> 
> It used to report being out of chain frames every time it happened,
> which
> wound up being too much.  You're right, doing it once might be good.
> 
> >    It looks like changing the related #define is the only way.
> 
> Yes, that is currently the only way.  Yours is by far the largest setup
> I've seen so far.  I've run the driver with 60 drives attached.
> 
> >    Does anyone have any experience with tuning this driver for high
> > throughput/large disk arrays? The shelves are all dual pathed, and
> with
> > the new gmultipath active/active support, I've still only been able to
> > achieve about 500MBytes per second across the controllers/drives.
> 
> Once you bump up the number of chain frames to the point where you
> aren't
> running out, I doubt the driver will be the big bottleneck.  It'll
> probably
> be other things higher up the stack.
> 
> > ps: I currently have a ccd on top of these drives which seems to
> > perform more consistenty then zfs. But that's an email for a different
> > day :-)
> 
> What sort of ZFS topology did you try?
> 
> I know for raidz2, and perhaps for raidz, ZFS is faster if your number
> of
> data disks is a power of 2.
> 
> If you want raidz2 protection, try creating arrays in groups of 10, so
> you
> wind up having 8 data disks.
> 
> Ken
> --
> Kenneth Merry
> ken at FreeBSD.ORG
> _______________________________________________
> freebsd-scsi at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-scsi
> To unsubscribe, send any mail to "freebsd-scsi-unsubscribe at freebsd.org"