Max Queue depth of HBA limited to 256 ?
Desai, Kashyap
Kashyap.Desai at lsi.com
Mon Mar 18 18:59:08 UTC 2013
Ken:
Thanks for your input. I switch back to this exercise today and found that mps driver is able to send more than 256 outstanding IO using some different benchmark tools like sysbench. (any tools which are thread based)
It is still not known to me that why I was not able to send more than 256 outstanding IOs if I use "dd" or "rawio".
Anyways that is not main concern for me.
Thanks for helping me on my query.
` Kashyap
> -----Original Message-----
> From: Kenneth D. Merry [mailto:ken at freebsd.org]
> Sent: Wednesday, February 06, 2013 2:47 AM
> To: Desai, Kashyap
> Cc: freebsd-scsi at freebsd.org; jhb at freebsd.org; McConnell, Stephen
> Subject: Re: Max Queue depth of HBA limited to 256 ?
>
>
> I'm able to get more than 255 commands outstanding to the controller in
> my configuration.
>
> For example:
>
> dev.mps.0.%desc: LSI SAS2116
> dev.mps.0.%driver: mps
> dev.mps.0.%location: slot=6 function=0 handle=\_SB_.PCI0.S30_
> dev.mps.0.%pnpinfo: vendor=0x1000 device=0x0064 subvendor=0x1000
> subdevice=0x30c0 class=0x010700
> dev.mps.0.%parent: pci0
> dev.mps.0.debug_level: 4
> dev.mps.0.disable_msix: 0
> dev.mps.0.disable_msi: 0
> dev.mps.0.firmware_version: 13.00.01.00
> dev.mps.0.driver_version: 14.00.00.01-fbsd
> dev.mps.0.io_cmds_active: 442
> dev.mps.0.io_cmds_highwater: 464
> dev.mps.0.chain_free: 354
> dev.mps.0.chain_free_lowwater: 181
> dev.mps.0.max_chains: 2048
> dev.mps.0.chain_alloc_fail: 0
>
> io_cmds_highwater is 464. Can you get more than 255 commands
> outstanding if you use more than 1 target?
>
> This is with 272 'dd' processes doing 1MB reads to 16 2TB and 3TB SAS
> drives behind 2 3Gb Maxim expanders:
>
> <SEAGATE ST32000444SS 0006> at scbus2 target 144 lun 0
> (pass4,sg4,da0)
> <SEAGATE ST32000444SS 0006> at scbus2 target 145 lun 0
> (pass5,sg5,da1)
> <SEAGATE ST33000650SS 0003> at scbus2 target 146 lun 0
> (pass6,sg6,da2)
> <SEAGATE ST33000650SS 0003> at scbus2 target 147 lun 0
> (pass7,sg7,da3)
> <SEAGATE ST32000444SS 0006> at scbus2 target 148 lun 0
> (pass8,sg8,da4)
> <SEAGATE ST32000444SS 0006> at scbus2 target 149 lun 0
> (pass9,sg9,da5)
> <SEAGATE ST33000650SS 0003> at scbus2 target 150 lun 0
> (pass10,sg10,da6)
> <SEAGATE ST33000650SS 0003> at scbus2 target 151 lun 0
> (pass11,sg11,da7)
> <SEAGATE ST32000444SS 0006> at scbus2 target 152 lun 0
> (pass12,sg12,da8)
> <SEAGATE ST32000444SS 0006> at scbus2 target 153 lun 0
> (pass13,sg13,da9)
> <SEAGATE ST32000444SS 0006> at scbus2 target 154 lun 0
> (pass14,sg14,da10)
> <SEAGATE ST32000444SS 0006> at scbus2 target 155 lun 0
> (pass15,sg15,da11)
> <SEAGATE ST33000650SS 0003> at scbus2 target 156 lun 0
> (pass16,sg16,da12)
> <SEAGATE ST33000650SS 0003> at scbus2 target 157 lun 0
> (pass17,sg17,da13)
> <SEAGATE ST33000650SS 0003> at scbus2 target 158 lun 0
> (pass18,sg18,da14)
> <SEAGATE ST33000650SS 0003> at scbus2 target 159 lun 0
> (pass19,sg19,da15)
>
> i.e. 17 iterations of this:
>
> ((i=0)); while [ $i -le 15 ]; do dd if=/dev/da$i of=/dev/null bs=1m &
> ((i++)); done
>
> The individual drives see varying numbers of tags, but nowhere near the
> maximum:
>
> [root at storage-domain ~]# camcontrol tags da15 -v
> (pass19:mps0:0:159:0): dev_openings 230
> (pass19:mps0:0:159:0): dev_active 25
> (pass19:mps0:0:159:0): devq_openings 230
> (pass19:mps0:0:159:0): devq_queued 0
> (pass19:mps0:0:159:0): held 0
> (pass19:mps0:0:159:0): mintags 2
> (pass19:mps0:0:159:0): maxtags 255
>
> What kind of drive is the target?
>
> Ken
>
> On Wed, Jan 23, 2013 at 00:44:31 +0530, Desai, Kashyap wrote:
> > LSI h/w needs more outstanding command in FW to get better Perf counts
> compare to other OS.
> >
> > Please suggest if whatever I have been observed is limitation from
> FreeBSD or we can tune it in Driver ?
> > My goals is to pump ~1000 outstanding IOs to the HBA. I see that it
> never goes beyond 255.
> >
> > Thanks,
> > Kashyap
> >
> > > -----Original Message-----
> > > From: owner-freebsd-scsi at freebsd.org [mailto:owner-freebsd-
> > > scsi at freebsd.org] On Behalf Of Desai, Kashyap
> > > Sent: Monday, January 21, 2013 11:18 PM
> > > To: Kenneth D. Merry
> > > Cc: freebsd-scsi at freebsd.org; jhb at freebsd.org; McConnell, Stephen
> > > Subject: RE: Max Queue depth of HBA limited to 256 ?
> > >
> > >
> > >
> > > > -----Original Message-----
> > > > From: Kenneth D. Merry [mailto:ken at freebsd.org]
> > > > Sent: Monday, January 21, 2013 10:35 PM
> > > > To: Desai, Kashyap
> > > > Cc: freebsd-scsi at freebsd.org; McConnell, Stephen; Saxena, Sumit;
> > > > jhb at freebsd.org
> > > > Subject: Re: Max Queue depth of HBA limited to 256 ?
> > > >
> > > > On Mon, Jan 21, 2013 at 20:15:47 +0530, Desai, Kashyap wrote:
> > > > > Hi,
> > > > >
> > > > > I was trying to check few things on LSI controller, where we
> > > > > have more
> > > > than 256 queue depth support.
> > > > > I added default maxtags in scsi/scsi_xpt.c as below. (Because I
> > > > > don't
> > > > want mattags to restrict any outstanding commands the LSI HBA.
> > > > >
> > > > > {
> > > > > /* Default tagged queuing parameters for all devices */
> > > > > {
> > > > > T_ANY, SIP_MEDIA_REMOVABLE|SIP_MEDIA_FIXED,
> > > > > /*vendor*/"*", /*product*/"*", /*revision*/"*"
> > > > > },
> > > > > /*quirks*/0, /*mintags*/2, /*maxtags*/1024 <---
> Default
> > > > maxtags were 256. I increase it to 10234
> > > > > },
> > > > >
> > > > >
> > > > > LSI's SAS-HBA and MR-HBA can support more than 256 outstanding
> > > > commands in Firmware. But due to some reason, I am not able to
> > > > pump more than 256 outstanding commands to the HBA.
> > > > >
> > > > > I used "rawio -p 256 /dev/da1" and more /dev/dax in loop. I have
> > > > sysctl parameter in Driver to display outstanding "FW commands".
> > > > Max value for FW outstanding only goes up to 256.
> > > > >
> > > > > Also from some other mail thread Subject "mfi driver
> > > > > performance", I
> > > > found that folks talk about tuning queue depth _but_ nobody
> > > > discussed to increase it beyond 256. Is there any limitation in
> FreeBSD ?
> > > > >
> > > >
> > > > As Jim pointed out, one thing to check is the values passed into
> > > > cam_sim_alloc(). In the case of the mps(4) driver, the
> > > > calculation is in mps_attach():
> > > >
> > > > sc->num_reqs = MIN(MPS_REQ_FRAMES, sc->facts->RequestCredit);
> > > >
> > > > What is reported for the RequestCredit on this particular adapter?
> > > >
> > > > The other question is, what does 'camcontrol tags daX -v' show
> > > > when you are running the test?
> > >
> > > Below is output of camcontrol tags da1 -v.
> > >
> > > dhcp-135-24-192-127# camcontrol tags da13 -v
> > > (pass13:mrsas0:0:13:0): dev_openings 1024
> > > (pass13:mrsas0:0:13:0): dev_active 0
> > > (pass13:mrsas0:0:13:0): devq_openings 1024
> > > (pass13:mrsas0:0:13:0): devq_queued 0
> > > (pass13:mrsas0:0:13:0): held 0
> > > (pass13:mrsas0:0:13:0): mintags 2
> > > (pass13:mrsas0:0:13:0): maxtags 1024
> > > dhcp-135-24-192-127# camcontrol tags da1 -v
> > > (pass1:mrsas0:0:1:0): dev_openings 1024
> > > (pass1:mrsas0:0:1:0): dev_active 0
> > > (pass1:mrsas0:0:1:0): devq_openings 1024
> > > (pass1:mrsas0:0:1:0): devq_queued 0
> > > (pass1:mrsas0:0:1:0): held 0
> > > (pass1:mrsas0:0:1:0): mintags 2
> > > (pass1:mrsas0:0:1:0): maxtags 1024
> > >
> > > Value 1024 is hard coded for my testing. In MegaRaid controller and
> > > SAS- HBA Driver read max commands value from FW.
> > > Similar to "RequestCredit".. Different FW has different value, but
> > > they are every time above 255.
> > >
> > >
> > > When I run IOs dev_active stays in range of 0-255 only. See below
> > > output when I run IOs on /dev/da1 and /dev/da13. I expect total
> > > dev_openings should go beyond 255, which is not happening.
> > >
> > >
> > > dhcp-135-24-192-127# camcontrol tags da1 -v
> > > (pass1:mrsas0:0:1:0): dev_openings 832
> > > (pass1:mrsas0:0:1:0): dev_active 192
> > > (pass1:mrsas0:0:1:0): devq_openings 832
> > > (pass1:mrsas0:0:1:0): devq_queued 0
> > > (pass1:mrsas0:0:1:0): held 0
> > > (pass1:mrsas0:0:1:0): mintags 2
> > > (pass1:mrsas0:0:1:0): maxtags 1024
> > > dhcp-135-24-192-127# camcontrol tags da13 -v
> > > (pass13:mrsas0:0:13:0): dev_openings 881
> > > (pass13:mrsas0:0:13:0): dev_active 143
> > > (pass13:mrsas0:0:13:0): devq_openings 881
> > > (pass13:mrsas0:0:13:0): devq_queued 0
> > > (pass13:mrsas0:0:13:0): held 0
> > > (pass13:mrsas0:0:13:0): mintags 2
> > > (pass13:mrsas0:0:13:0): maxtags 1024
> > >
> > >
> > >
> > >
> > > Jim:
> > > Below is my API call. I have hard code value "queue_depth" = 1024
> > >
> > > sc->sim_0 = cam_sim_alloc(mrsas_action, mrsas_poll, "mrsas", sc,
> > > device_get_unit(sc->mrsas_dev), &sc->sim_lock, queue_depth,
> > > queue_depth, devq);
> > >
> > > ~ Kashyap
> > >
> > > >
> > > > Ken
> > > > --
> > > > Kenneth Merry
> > > > ken at FreeBSD.ORG
> > > _______________________________________________
> > > freebsd-scsi at freebsd.org mailing list
> > > http://lists.freebsd.org/mailman/listinfo/freebsd-scsi
> > > To unsubscribe, send any mail to "freebsd-scsi-
> unsubscribe at freebsd.org"
>
> --
> Kenneth Merry
> ken at FreeBSD.ORG
More information about the freebsd-scsi
mailing list