LSI2008 controller clobbers first disk with new LSI mps driver

Fri Feb 17 08:48:55 UTC 2012

Kashyap,

Yes I've run all 4 drivers on the same machine with the 10.00.02.00 FW:

Old mps from (I believe) FreeBSD developers:

mps0: <LSI SAS2008> port 0xe000-0xe0ff mem
0xfbd3c000-0xfbd3ffff,0xfbd40000-0xfbd7ffff irq 26 at device 0.0 on pci4
mps0: Firmware: 10.00.02.00
mps0: IOCCapabilities:
1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
mps0: [ITHREAD]
da0 at mps0 bus 0 scbus0 target 1 lun 0
da1 at mps0 bus 0 scbus0 target 2 lun 0
da2 at mps0 bus 0 scbus0 target 3 lun 0
da3 at mps0 bus 0 scbus0 target 4 lun 0
da4 at mps0 bus 0 scbus0 target 5 lun 0
da5 at mps0 bus 0 scbus0 target 6 lun 0
da6 at mps0 bus 0 scbus0 target 7 lun 0
da8 at mps0 bus 0 scbus0 target 9 lun 0
da7 at mps0 bus 0 scbus0 target 8 lun 0
da10 at mps0 bus 0 scbus0 target 11 lun 0
da9 at mps0 bus 0 scbus0 target 10 lun 0
da11 at mps0 bus 0 scbus0 target 12 lun 0
ses0 at mps0 bus 0 scbus0 target 13 lun 0
ses0: <LSI CORP SAS2X28 0717> Fixed Enclosure Services SCSI-5 device
ses0: 600.000MB/s transfers
ses0: Command Queueing enabled
ses0: SCSI-3 SES Device

mpslsi.ko binary from 2011:

mpslsi0: <LSI SAS2008> port 0xe000-0xe0ff mem
0xfbd3c000-0xfbd3ffff,0xfbd40000-0xfbd7ffff irq 26 at device 0.0 on pci4
mpslsi0: Firmware: 10.00.02.00, Driver: 08.55.00.00
mpslsi0: IOCCapabilities:
1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
mpslsi0: [ITHREAD]
ses0 at mpslsi0 bus 0 target 8 lun 0
ses0: <LSI CORP SAS2X28 0717> Fixed Enclosure Services SCSI-5 device
ses0: 600.000MB/s transfers
ses0: Command Queueing Enabled
ses0: SCSI-3 SES Device
da0 at mpslsi0 bus 0 target 9 lun 0
da1 at mpslsi0 bus 0 target 10 lun 0
da2 at mpslsi0 bus 0 target 11 lun 0
da3 at mpslsi0 bus 0 target 12 lun 0
da4 at mpslsi0 bus 0 target 13 lun 0
da5 at mpslsi0 bus 0 target 14 lun 0
da6 at mpslsi0 bus 0 target 15 lun 0
da7 at mpslsi0 bus 0 target 16 lun 0
da8 at mpslsi0 bus 0 target 17 lun 0
da9 at mpslsi0 bus 0 target 18 lun 0
da10 at mpslsi0 bus 0 target 19 lun 0

11.255.03.00-fbsd:

mps0: <LSI SAS2008> port 0xe000-0xe0ff mem
0xfbd3c000-0xfbd3ffff,0xfbd40000-0xfbd7ffff irq 26 at device 0.0 on pci4
mps0: Firmware: 10.00.02.00, Driver: 11.255.03.00-fbsd
mps0: IOCCapabilities:
1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
mps0: [ITHREAD]
ses0 at mps0 bus 0 scbus0 target 8 lun 0
ses0: <LSI CORP SAS2X28 0717> Fixed Enclosure Services SCSI-5 device
ses0: 600.000MB/s transfers
ses0: Command Queueing enabled
ses0: SCSI-3 SES Device
da0 at mps0 bus 0 scbus0 target 9 lun 0
da1 at mps0 bus 0 scbus0 target 10 lun 0
da2 at mps0 bus 0 scbus0 target 11 lun 0
da3 at mps0 bus 0 scbus0 target 12 lun 0
da4 at mps0 bus 0 scbus0 target 13 lun 0
da5 at mps0 bus 0 scbus0 target 14 lun 0
da6 at mps0 bus 0 scbus0 target 15 lun 0
da7 at mps0 bus 0 scbus0 target 16 lun 0
da8 at mps0 bus 0 scbus0 target 17 lun 0
da9 at mps0 bus 0 scbus0 target 18 lun 0
da10 at mps0 bus 0 scbus0 target 19 lun 0

Current 13.00.00.00-fbsd:

mps0: <LSI SAS2008> port 0xe000-0xe0ff mem
0xfbd3c000-0xfbd3ffff,0xfbd40000-0xfbd7ffff irq 26 at device 0.0 on pci4
mps0: Firmware: 10.00.02.00, Driver: 13.00.00.00-fbsd
mps0: IOCCapabilities:
1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
mps0: [ITHREAD]
da1 at mps0 bus 0 scbus0 target 10 lun 0
da0 at mps0 bus 0 scbus0 target 9 lun 0
da2 at mps0 bus 0 scbus0 target 11 lun 0
da4 at mps0 bus 0 scbus0 target 13 lun 0
da5 at mps0 bus 0 scbus0 target 14 lun 0
da3 at mps0 bus 0 scbus0 target 12 lun 0
da7 at mps0 bus 0 scbus0 target 16 lun 0
da8 at mps0 bus 0 scbus0 target 17 lun 0
da6 at mps0 bus 0 scbus0 target 15 lun 0
da9 at mps0 bus 0 scbus0 target 18 lun 0
da10 at mps0 bus 0 scbus0 target 19 lun 0
ses0 at mps0 bus 0 scbus0 target 8 lun 0
ses0: <LSI CORP SAS2X28 0717> Fixed Enclosure Services SCSI-5 device
ses0: 600.000MB/s transfers
ses0: Command Queueing enabled
ses0: SCSI-3 SES Device

Jason

On Fri, Feb 17, 2012 at 1:24 AM, Desai, Kashyap <Kashyap.Desai at lsi.com>wrote:

> Jason,
>
> Me also surprised when I see so many queries on <mps>.
> Really wonderful to know that there are good amount of <mpslsi> driver use
> as well.
>
> I was under impression that if you keep same FW and just change Driver,
> there should not be any difference in Target IDs assigned to Device
> connected behind that HBA.
>
> Is this possible for you to keep everything unchanged and just change
> Driver version and see how things behaves. Please share "dmesg" logs of
> your both drivers.  BTW, did you tested "13.00.00.00-fbsd" and
> "11.00.00.00"  on same machine ?
>
> FYI: We never observe this kind of issue in our lab.
>
> ` Kashyap
>
> From: Jason Wolfe [mailto:nitroboost at gmail.com]
> Sent: Friday, February 17, 2012 1:31 PM
> To: Desai, Kashyap
> Cc: freebsd-scsi at freebsd.org
> Subject: Re: LSI2008 controller clobbers first disk with new LSI mps driver
>
> Kashyap,
>
> Ah a response from LSI, that's a pleasant surprise :)  Everything you've
> stated looks correct to me, the FreeBSD developed driver that has been
> replaced by the LSI driver has no issues with either firmware.  Your likely
> aware, but just to confirm, here is the history of the 3 various LSI
> drivers that have the issue on the 10.00.02.00 FW:
>
> 11.00.00.00 - binary driver I had received from you guys in mid  2011,
> mpslsi.ko, one for each 7.2-RELEASE and 8.2-RELEASE
>
> 11.255.03.00-fbsd - initial LSI driver committed to 8-STABLE on 2/2,
> r230922
>
> 13.00.00.00-fbsd - commited to 8-STABLE on 2/14, r231680
>
> I have about 40 boxes with the 10.00.02.00 FW I've tested, so I'm fairly
> certain it's not bad hardware or a fluke.  You guys haven't seen anything
> like this in house?  I'd hate to hear I have to update the FW on these
> boxes as they are all quite a ways from me, though it seems there is some
> way to work around the behavior in the driver as the FreeBSD one does?  I
> have a few of these boxes out of service so I'm game to try some things out
> should that help.
>
> Thank for the response,
> Jason
>
> On Thu, Feb 16, 2012 at 11:29 PM, Desai, Kashyap <Kashyap.Desai at lsi.com>
> wrote:
> Jason,
>
> I have gone through your data provided in this thread. It is well
> understood because of your descriptive data.
>
> So What I understood here is:
>
> 1. You tested with HBA Fw "07.00.00.00" and "10.00.02.00"
>
> 2. you have run your test on two different LSI BIOS versions.
> Grabbed from below line.
> MPT2BIOS-7.11.00.00 (2010.07.29) / PRODUCT REVISION 7.00.00.00
> MPT2BIOS-7.19.00.00 (2011.05.16) / PRODUCT REVISION 10.00.02.00
>
> Now I am able to see below three difference in your setup.
>
> See FW version and check starting target id, all three has different way
> of assigning TargetIDs.
> For first two case target id start with "8" but SES device assignment is
> different.
> Last case target id start with "1"
>
>
> mps0: Firmware: 07.00.00.00, Driver: 11.255.03.00-fbsd (OR)
> 13.00.00.00-fbsd
> > > <SEAGATE ST91000640SS 0001> at scbus0 target 8 lun 0 (pass0,da0)
> > > <SEAGATE ST91000640SS 0001> at scbus0 target 9 lun 0 (pass1,da1)
> > > <SEAGATE ST91000640SS 0001> at scbus0 target 10 lun 0 (pass2,da2)
> > > <SEAGATE ST91000640SS 0001> at scbus0 target 11 lun 0 (pass3,da3)
> > > <SEAGATE ST91000640SS 0001> at scbus0 target 12 lun 0 (pass4,da4)
> > > <SEAGATE ST91000640SS 0001> at scbus0 target 13 lun 0 (pass5,da5)
> > > <SEAGATE ST91000640SS 0001> at scbus0 target 14 lun 0 (pass6,da6)
> > > <SEAGATE ST91000640SS 0001> at scbus0 target 15 lun 0 (pass7,da7)
> > > <SEAGATE ST91000640SS 0001> at scbus0 target 16 lun 0 (pass8,da8)
> > > <SEAGATE ST91000640SS 0001> at scbus0 target 17 lun 0 (pass9,da9)
> > > <SEAGATE ST91000640SS 0001> at scbus0 target 18 lun 0 (pass10,da10)
> > > <SEAGATE ST91000640SS 0001> at scbus0 target 19 lun 0 (pass11,da11)
> > > <LSI CORP SAS2X28 0717> at scbus0 target 20 lun 0 (ses0,pass12)
>
> mps0: Firmware: 10.00.02.00, Driver: 13.00.00.00-fbsd
> mps0: Firmware: 10.00.02.00, Driver: 11.00.00.00 (OR) 8.2-STABLE Inbox
> > > <LSI CORP SAS2X28 0717> at scbus0 target 8 lun 0 (ses0,pass0)
> > > <SEAGATE ST91000640SS 0001> at scbus0 target 9 lun 0 (da0,pass1)
> > > <SEAGATE ST91000640SS 0001> at scbus0 target 10 lun 0 (da1,pass2)
> > > <SEAGATE ST91000640SS 0001> at scbus0 target 11 lun 0 (da2,pass3)
> > > <SEAGATE ST91000640SS 0001> at scbus0 target 12 lun 0 (da3,pass4)
> > > <SEAGATE ST91000640SS 0001> at scbus0 target 13 lun 0 (da4,pass5)
> > > <SEAGATE ST91000640SS 0001> at scbus0 target 14 lun 0 (da5,pass6)
> > > <SEAGATE ST91000640SS 0001> at scbus0 target 15 lun 0 (da6,pass7)
> > > <SEAGATE ST91000640SS 0001> at scbus0 target 16 lun 0 (da7,pass8)
> > > <SEAGATE ST91000640SS 0001> at scbus0 target 17 lun 0 (da8,pass9)
> > > <SEAGATE ST91000640SS 0001> at scbus0 target 18 lun 0 (da9,pass10)
> > > <SEAGATE ST91000640SS 0001> at scbus0 target 19 lun 0 (da10,pass11)
>
> On the FBSD developed driver active in 8-STABLE prior to the LSI
> Release (Firmware: 10.00.02.00)
>
> > > <SEAGATE ST91000640SS 0001> at scbus0 target 1 lun 0 (pass0,da0)
> > > <SEAGATE ST91000640SS 0001> at scbus0 target 2 lun 0 (pass1,da1)
> > > <SEAGATE ST91000640SS 0001> at scbus0 target 3 lun 0 (pass2,da2)
> > > <SEAGATE ST91000640SS 0001> at scbus0 target 4 lun 0 (pass3,da3)
> > > <SEAGATE ST91000640SS 0001> at scbus0 target 5 lun 0 (pass4,da4)
> > > <SEAGATE ST91000640SS 0001> at scbus0 target 6 lun 0 (pass5,da5)
> > > <SEAGATE ST91000640SS 0001> at scbus0 target 7 lun 0 (pass6,da6)
> > > <SEAGATE ST91000640SS 0001> at scbus0 target 8 lun 0 (pass7,da7)
> > > <SEAGATE ST91000640SS 0001> at scbus0 target 9 lun 0 (pass8,da8)
> > > <SEAGATE ST91000640SS 0001> at scbus0 target 10 lun 0 (pass9,da9)
> > > <SEAGATE ST91000640SS 0001> at scbus0 target 11 lun 0 (pass10,da10)
> > > <SEAGATE ST91000640SS 0001> at scbus0 target 12 lun 0 (pass11,da11)
> > > <LSI CORP SAS2X28 0717> at scbus0 target 13 lun 0 (ses0,pass12)
>
>
> In summary,  (please confirm)
> 1. you have not seen any issue if you use "07.00.00.00" FW version.
> 2. _but_ when you use "10.00.02.00" FW, with "13.00.00.00-fbsd" driver
> version you are seeing
> SES is detected before Drives as pass0.
> 3. When you use "10.00.02.00" FW with 8-STABLE inbox FBSD driver, you are
> finding SES device detected after Drives.
>
>
> All driver is doing here is asking CAM layer to scan Bus when there is any
> device added on that bus.
> So depending upon actual target Id  assigned by FW, it will be detected to
> camcontrol.
>
> So bottom line is FW plays major role in sequencing Drives behind LSI
> controller.
>
>  ~ Kashyap
>