Max Queue depth of HBA limited to 256 ?
Desai, Kashyap
Kashyap.Desai at lsi.com
Mon Feb 11 16:12:20 UTC 2013
Ken:
I am still not able to pump more than 256 command. (This time I simulated similar to your setup. I tried <mps> driver)
For example:
dev.mps.0.%desc: LSI SAS2008
dev.mps.0.%driver: mps
dev.mps.0.%location: slot=0 function=0
dev.mps.0.%pnpinfo: vendor=0x1000 device=0x0072 subvendor=0x1000 subdevice=0x0072 class=0x010700
dev.mps.0.%parent: pci2
dev.mps.0.debug_level: 0
dev.mps.0.disable_msix: 0
dev.mps.0.disable_msi: 0
dev.mps.0.firmware_version: 14.00.00.00
dev.mps.0.driver_version: 14.00.00.01-fbsd
dev.mps.0.io_cmds_active: 256
dev.mps.0.io_cmds_highwater: 256
dev.mps.0.chain_free: 2048
dev.mps.0.chain_free_lowwater: 2047
dev.mps.0.max_chains: 2048
dev.mps.0.chain_alloc_fail: 0
io_cmds_highwater = 256
Here is the system information:
<SEAGATE ST9500620SS AS05> at scbus0 target 10 lun 0 (pass3,da1)
<SEAGATE ST9500620SS AS05> at scbus0 target 11 lun 0 (pass4,da2)
<SEAGATE ST9500620SS AS05> at scbus0 target 12 lun 0 (pass5,da3)
<SEAGATE ST9500620SS AS05> at scbus0 target 13 lun 0 (pass2,da0)
<LSI CORP Bobcat 0c00> at scbus0 target 14 lun 0 (pass6,ses0) -- Expander
Running 128 dd on each target using small shell scripts:
---------------------------------------------------------
#!/bin/sh
max=128
for i in `seq 1 $max`
do
dd if=/dev/da0 of=/dev/null bs=1m &
dd if=/dev/da1 of=/dev/null bs=1m &
dd if=/dev/da2 of=/dev/null bs=1m &
dd if=/dev/da3 of=/dev/null bs=1m &
echo "$i"
done
---------------------------------------------------------------------------------------------
The individual drives logs:
dhcp-135-24-192-146# camcontrol tags da3 -v
(pass5:mps0:0:12:0): dev_openings 248
(pass5:mps0:0:12:0): dev_active 7
(pass5:mps0:0:12:0): devq_openings 248
(pass5:mps0:0:12:0): devq_queued 0
(pass5:mps0:0:12:0): held 0
(pass5:mps0:0:12:0): mintags 2
(pass5:mps0:0:12:0): maxtags 255
dhcp-135-24-192-146# camcontrol tags da2 -v
(pass4:mps0:0:11:0): dev_openings 155
(pass4:mps0:0:11:0): dev_active 100
(pass4:mps0:0:11:0): devq_openings 155
(pass4:mps0:0:11:0): devq_queued 0
(pass4:mps0:0:11:0): held 0
(pass4:mps0:0:11:0): mintags 2
(pass4:mps0:0:11:0): maxtags 255
dhcp-135-24-192-146# camcontrol tags da1 -v
(pass3:mps0:0:10:0): dev_openings 145
(pass3:mps0:0:10:0): dev_active 110
(pass3:mps0:0:10:0): devq_openings 145
(pass3:mps0:0:10:0): devq_queued 0
(pass3:mps0:0:10:0): held 0
(pass3:mps0:0:10:0): mintags 2
(pass3:mps0:0:10:0): maxtags 255
dhcp-135-24-192-146# camcontrol tags da0 -v
(pass2:mps0:0:13:0): dev_openings 233
(pass2:mps0:0:13:0): dev_active 22
(pass2:mps0:0:13:0): devq_openings 233
(pass2:mps0:0:13:0): devq_queued 0
(pass2:mps0:0:13:0): held 0
(pass2:mps0:0:13:0): mintags 2
(pass2:mps0:0:13:0): maxtags 255
Is there any change in latest Upstream kernel ? Mine is not very much latest FreeBSD OS..!
I will try with latest upstream freebsd OS.
` Kashyap
> -----Original Message-----
> From: Kenneth D. Merry [mailto:ken at freebsd.org]
> Sent: Wednesday, February 06, 2013 2:47 AM
> To: Desai, Kashyap
> Cc: freebsd-scsi at freebsd.org; jhb at freebsd.org; McConnell, Stephen
> Subject: Re: Max Queue depth of HBA limited to 256 ?
>
>
> I'm able to get more than 255 commands outstanding to the controller in
> my configuration.
>
> For example:
>
> dev.mps.0.%desc: LSI SAS2116
> dev.mps.0.%driver: mps
> dev.mps.0.%location: slot=6 function=0 handle=\_SB_.PCI0.S30_
> dev.mps.0.%pnpinfo: vendor=0x1000 device=0x0064 subvendor=0x1000
> subdevice=0x30c0 class=0x010700
> dev.mps.0.%parent: pci0
> dev.mps.0.debug_level: 4
> dev.mps.0.disable_msix: 0
> dev.mps.0.disable_msi: 0
> dev.mps.0.firmware_version: 13.00.01.00
> dev.mps.0.driver_version: 14.00.00.01-fbsd
> dev.mps.0.io_cmds_active: 442
> dev.mps.0.io_cmds_highwater: 464
> dev.mps.0.chain_free: 354
> dev.mps.0.chain_free_lowwater: 181
> dev.mps.0.max_chains: 2048
> dev.mps.0.chain_alloc_fail: 0
>
> io_cmds_highwater is 464. Can you get more than 255 commands
> outstanding if you use more than 1 target?
>
> This is with 272 'dd' processes doing 1MB reads to 16 2TB and 3TB SAS
> drives behind 2 3Gb Maxim expanders:
>
> <SEAGATE ST32000444SS 0006> at scbus2 target 144 lun 0
> (pass4,sg4,da0)
> <SEAGATE ST32000444SS 0006> at scbus2 target 145 lun 0
> (pass5,sg5,da1)
> <SEAGATE ST33000650SS 0003> at scbus2 target 146 lun 0
> (pass6,sg6,da2)
> <SEAGATE ST33000650SS 0003> at scbus2 target 147 lun 0
> (pass7,sg7,da3)
> <SEAGATE ST32000444SS 0006> at scbus2 target 148 lun 0
> (pass8,sg8,da4)
> <SEAGATE ST32000444SS 0006> at scbus2 target 149 lun 0
> (pass9,sg9,da5)
> <SEAGATE ST33000650SS 0003> at scbus2 target 150 lun 0
> (pass10,sg10,da6)
> <SEAGATE ST33000650SS 0003> at scbus2 target 151 lun 0
> (pass11,sg11,da7)
> <SEAGATE ST32000444SS 0006> at scbus2 target 152 lun 0
> (pass12,sg12,da8)
> <SEAGATE ST32000444SS 0006> at scbus2 target 153 lun 0
> (pass13,sg13,da9)
> <SEAGATE ST32000444SS 0006> at scbus2 target 154 lun 0
> (pass14,sg14,da10)
> <SEAGATE ST32000444SS 0006> at scbus2 target 155 lun 0
> (pass15,sg15,da11)
> <SEAGATE ST33000650SS 0003> at scbus2 target 156 lun 0
> (pass16,sg16,da12)
> <SEAGATE ST33000650SS 0003> at scbus2 target 157 lun 0
> (pass17,sg17,da13)
> <SEAGATE ST33000650SS 0003> at scbus2 target 158 lun 0
> (pass18,sg18,da14)
> <SEAGATE ST33000650SS 0003> at scbus2 target 159 lun 0
> (pass19,sg19,da15)
>
> i.e. 17 iterations of this:
>
> ((i=0)); while [ $i -le 15 ]; do dd if=/dev/da$i of=/dev/null bs=1m &
> ((i++)); done
>
> The individual drives see varying numbers of tags, but nowhere near the
> maximum:
>
> [root at storage-domain ~]# camcontrol tags da15 -v
> (pass19:mps0:0:159:0): dev_openings 230
> (pass19:mps0:0:159:0): dev_active 25
> (pass19:mps0:0:159:0): devq_openings 230
> (pass19:mps0:0:159:0): devq_queued 0
> (pass19:mps0:0:159:0): held 0
> (pass19:mps0:0:159:0): mintags 2
> (pass19:mps0:0:159:0): maxtags 255
>
> What kind of drive is the target?
>
> Ken
>
> On Wed, Jan 23, 2013 at 00:44:31 +0530, Desai, Kashyap wrote:
> > LSI h/w needs more outstanding command in FW to get better Perf counts
> compare to other OS.
> >
> > Please suggest if whatever I have been observed is limitation from
> FreeBSD or we can tune it in Driver ?
> > My goals is to pump ~1000 outstanding IOs to the HBA. I see that it
> never goes beyond 255.
> >
> > Thanks,
> > Kashyap
> >
> > > -----Original Message-----
> > > From: owner-freebsd-scsi at freebsd.org [mailto:owner-freebsd-
> > > scsi at freebsd.org] On Behalf Of Desai, Kashyap
> > > Sent: Monday, January 21, 2013 11:18 PM
> > > To: Kenneth D. Merry
> > > Cc: freebsd-scsi at freebsd.org; jhb at freebsd.org; McConnell, Stephen
> > > Subject: RE: Max Queue depth of HBA limited to 256 ?
> > >
> > >
> > >
> > > > -----Original Message-----
> > > > From: Kenneth D. Merry [mailto:ken at freebsd.org]
> > > > Sent: Monday, January 21, 2013 10:35 PM
> > > > To: Desai, Kashyap
> > > > Cc: freebsd-scsi at freebsd.org; McConnell, Stephen; Saxena, Sumit;
> > > > jhb at freebsd.org
> > > > Subject: Re: Max Queue depth of HBA limited to 256 ?
> > > >
> > > > On Mon, Jan 21, 2013 at 20:15:47 +0530, Desai, Kashyap wrote:
> > > > > Hi,
> > > > >
> > > > > I was trying to check few things on LSI controller, where we
> > > > > have more
> > > > than 256 queue depth support.
> > > > > I added default maxtags in scsi/scsi_xpt.c as below. (Because I
> > > > > don't
> > > > want mattags to restrict any outstanding commands the LSI HBA.
> > > > >
> > > > > {
> > > > > /* Default tagged queuing parameters for all devices */
> > > > > {
> > > > > T_ANY, SIP_MEDIA_REMOVABLE|SIP_MEDIA_FIXED,
> > > > > /*vendor*/"*", /*product*/"*", /*revision*/"*"
> > > > > },
> > > > > /*quirks*/0, /*mintags*/2, /*maxtags*/1024 <---
> Default
> > > > maxtags were 256. I increase it to 10234
> > > > > },
> > > > >
> > > > >
> > > > > LSI's SAS-HBA and MR-HBA can support more than 256 outstanding
> > > > commands in Firmware. But due to some reason, I am not able to
> > > > pump more than 256 outstanding commands to the HBA.
> > > > >
> > > > > I used "rawio -p 256 /dev/da1" and more /dev/dax in loop. I have
> > > > sysctl parameter in Driver to display outstanding "FW commands".
> > > > Max value for FW outstanding only goes up to 256.
> > > > >
> > > > > Also from some other mail thread Subject "mfi driver
> > > > > performance", I
> > > > found that folks talk about tuning queue depth _but_ nobody
> > > > discussed to increase it beyond 256. Is there any limitation in
> FreeBSD ?
> > > > >
> > > >
> > > > As Jim pointed out, one thing to check is the values passed into
> > > > cam_sim_alloc(). In the case of the mps(4) driver, the
> > > > calculation is in mps_attach():
> > > >
> > > > sc->num_reqs = MIN(MPS_REQ_FRAMES, sc->facts->RequestCredit);
> > > >
> > > > What is reported for the RequestCredit on this particular adapter?
> > > >
> > > > The other question is, what does 'camcontrol tags daX -v' show
> > > > when you are running the test?
> > >
> > > Below is output of camcontrol tags da1 -v.
> > >
> > > dhcp-135-24-192-127# camcontrol tags da13 -v
> > > (pass13:mrsas0:0:13:0): dev_openings 1024
> > > (pass13:mrsas0:0:13:0): dev_active 0
> > > (pass13:mrsas0:0:13:0): devq_openings 1024
> > > (pass13:mrsas0:0:13:0): devq_queued 0
> > > (pass13:mrsas0:0:13:0): held 0
> > > (pass13:mrsas0:0:13:0): mintags 2
> > > (pass13:mrsas0:0:13:0): maxtags 1024
> > > dhcp-135-24-192-127# camcontrol tags da1 -v
> > > (pass1:mrsas0:0:1:0): dev_openings 1024
> > > (pass1:mrsas0:0:1:0): dev_active 0
> > > (pass1:mrsas0:0:1:0): devq_openings 1024
> > > (pass1:mrsas0:0:1:0): devq_queued 0
> > > (pass1:mrsas0:0:1:0): held 0
> > > (pass1:mrsas0:0:1:0): mintags 2
> > > (pass1:mrsas0:0:1:0): maxtags 1024
> > >
> > > Value 1024 is hard coded for my testing. In MegaRaid controller and
> > > SAS- HBA Driver read max commands value from FW.
> > > Similar to "RequestCredit".. Different FW has different value, but
> > > they are every time above 255.
> > >
> > >
> > > When I run IOs dev_active stays in range of 0-255 only. See below
> > > output when I run IOs on /dev/da1 and /dev/da13. I expect total
> > > dev_openings should go beyond 255, which is not happening.
> > >
> > >
> > > dhcp-135-24-192-127# camcontrol tags da1 -v
> > > (pass1:mrsas0:0:1:0): dev_openings 832
> > > (pass1:mrsas0:0:1:0): dev_active 192
> > > (pass1:mrsas0:0:1:0): devq_openings 832
> > > (pass1:mrsas0:0:1:0): devq_queued 0
> > > (pass1:mrsas0:0:1:0): held 0
> > > (pass1:mrsas0:0:1:0): mintags 2
> > > (pass1:mrsas0:0:1:0): maxtags 1024
> > > dhcp-135-24-192-127# camcontrol tags da13 -v
> > > (pass13:mrsas0:0:13:0): dev_openings 881
> > > (pass13:mrsas0:0:13:0): dev_active 143
> > > (pass13:mrsas0:0:13:0): devq_openings 881
> > > (pass13:mrsas0:0:13:0): devq_queued 0
> > > (pass13:mrsas0:0:13:0): held 0
> > > (pass13:mrsas0:0:13:0): mintags 2
> > > (pass13:mrsas0:0:13:0): maxtags 1024
> > >
> > >
> > >
> > >
> > > Jim:
> > > Below is my API call. I have hard code value "queue_depth" = 1024
> > >
> > > sc->sim_0 = cam_sim_alloc(mrsas_action, mrsas_poll, "mrsas", sc,
> > > device_get_unit(sc->mrsas_dev), &sc->sim_lock, queue_depth,
> > > queue_depth, devq);
> > >
> > > ~ Kashyap
> > >
> > > >
> > > > Ken
> > > > --
> > > > Kenneth Merry
> > > > ken at FreeBSD.ORG
> > > _______________________________________________
> > > freebsd-scsi at freebsd.org mailing list
> > > http://lists.freebsd.org/mailman/listinfo/freebsd-scsi
> > > To unsubscribe, send any mail to "freebsd-scsi-
> unsubscribe at freebsd.org"
>
> --
> Kenneth Merry
> ken at FreeBSD.ORG
More information about the freebsd-scsi
mailing list