LSI2008 controller clobbers first disk with new LSI mps driver

Fri Feb 17 09:07:11 UTC 2012

Jason,

Can you switch your mail client to "text mode". It is difficult to do inline reply in html format.

<mpslsi> version 08.55.00.00, <mps> version 11.255.03.00-fbsd and <mps> version 13.00.00.00-fbsd detected SES device as Target Id = 8.
This target Id is retrieved from FW. I am able to map my theory in your case.

Now, Only problem is when you use "Old mps FreeBSD developers" things are different (and you would like to see same result using other driver versions ).
Here is a reason why behavior is different.
Our older mps driver does not had mapping code available in few driver releases. (see mps_mapping.c file)
Because of that reason Driver will assign Target Id sequential (Internal sequence) from Event received from FW.

Summary is all driver behavior with FW 10.00.02.00 is as expected.

Can you boot into verbose mode and send me log where it actual hangs ?  Also what is your /etc/fstab entry.

` Kashyap

From: Jason Wolfe [mailto:nitroboost at gmail.com]
Sent: Friday, February 17, 2012 2:19 PM
To: Desai, Kashyap
Cc: freebsd-scsi at freebsd.org
Subject: Re: LSI2008 controller clobbers first disk with new LSI mps driver

Kashyap,

Yes I've run all 4 drivers on the same machine with the 10.00.02.00 FW:

Old mps from (I believe) FreeBSD developers:

mps0: <LSI SAS2008> port 0xe000-0xe0ff mem 0xfbd3c000-0xfbd3ffff,0xfbd40000-0xfbd7ffff irq 26 at device 0.0 on pci4
mps0: Firmware: 10.00.02.00
mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
mps0: [ITHREAD]
da0 at mps0 bus 0 scbus0 target 1 lun 0
da1 at mps0 bus 0 scbus0 target 2 lun 0
da2 at mps0 bus 0 scbus0 target 3 lun 0
da3 at mps0 bus 0 scbus0 target 4 lun 0
da4 at mps0 bus 0 scbus0 target 5 lun 0
da5 at mps0 bus 0 scbus0 target 6 lun 0
da6 at mps0 bus 0 scbus0 target 7 lun 0
da8 at mps0 bus 0 scbus0 target 9 lun 0
da7 at mps0 bus 0 scbus0 target 8 lun 0
da10 at mps0 bus 0 scbus0 target 11 lun 0
da9 at mps0 bus 0 scbus0 target 10 lun 0
da11 at mps0 bus 0 scbus0 target 12 lun 0
ses0 at mps0 bus 0 scbus0 target 13 lun 0
ses0: <LSI CORP SAS2X28 0717> Fixed Enclosure Services SCSI-5 device
ses0: 600.000MB/s transfers
ses0: Command Queueing enabled
ses0: SCSI-3 SES Device

mpslsi.ko binary from 2011:

mpslsi0: <LSI SAS2008> port 0xe000-0xe0ff mem 0xfbd3c000-0xfbd3ffff,0xfbd40000-0xfbd7ffff irq 26 at device 0.0 on pci4
mpslsi0: Firmware: 10.00.02.00, Driver: 08.55.00.00
mpslsi0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
mpslsi0: [ITHREAD]
ses0 at mpslsi0 bus 0 target 8 lun 0
ses0: <LSI CORP SAS2X28 0717> Fixed Enclosure Services SCSI-5 device
ses0: 600.000MB/s transfers
ses0: Command Queueing Enabled
ses0: SCSI-3 SES Device
da0 at mpslsi0 bus 0 target 9 lun 0
da1 at mpslsi0 bus 0 target 10 lun 0
da2 at mpslsi0 bus 0 target 11 lun 0
da3 at mpslsi0 bus 0 target 12 lun 0
da4 at mpslsi0 bus 0 target 13 lun 0
da5 at mpslsi0 bus 0 target 14 lun 0
da6 at mpslsi0 bus 0 target 15 lun 0
da7 at mpslsi0 bus 0 target 16 lun 0
da8 at mpslsi0 bus 0 target 17 lun 0
da9 at mpslsi0 bus 0 target 18 lun 0
da10 at mpslsi0 bus 0 target 19 lun 0

11.255.03.00-fbsd:

mps0: <LSI SAS2008> port 0xe000-0xe0ff mem 0xfbd3c000-0xfbd3ffff,0xfbd40000-0xfbd7ffff irq 26 at device 0.0 on pci4
mps0: Firmware: 10.00.02.00, Driver: 11.255.03.00-fbsd
mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
mps0: [ITHREAD]
ses0 at mps0 bus 0 scbus0 target 8 lun 0
ses0: <LSI CORP SAS2X28 0717> Fixed Enclosure Services SCSI-5 device
ses0: 600.000MB/s transfers
ses0: Command Queueing enabled
ses0: SCSI-3 SES Device
da0 at mps0 bus 0 scbus0 target 9 lun 0
da1 at mps0 bus 0 scbus0 target 10 lun 0
da2 at mps0 bus 0 scbus0 target 11 lun 0
da3 at mps0 bus 0 scbus0 target 12 lun 0
da4 at mps0 bus 0 scbus0 target 13 lun 0
da5 at mps0 bus 0 scbus0 target 14 lun 0
da6 at mps0 bus 0 scbus0 target 15 lun 0
da7 at mps0 bus 0 scbus0 target 16 lun 0
da8 at mps0 bus 0 scbus0 target 17 lun 0
da9 at mps0 bus 0 scbus0 target 18 lun 0
da10 at mps0 bus 0 scbus0 target 19 lun 0

Current 13.00.00.00-fbsd:

mps0: <LSI SAS2008> port 0xe000-0xe0ff mem 0xfbd3c000-0xfbd3ffff,0xfbd40000-0xfbd7ffff irq 26 at device 0.0 on pci4
mps0: Firmware: 10.00.02.00, Driver: 13.00.00.00-fbsd
mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
mps0: [ITHREAD]
da1 at mps0 bus 0 scbus0 target 10 lun 0
da0 at mps0 bus 0 scbus0 target 9 lun 0
da2 at mps0 bus 0 scbus0 target 11 lun 0
da4 at mps0 bus 0 scbus0 target 13 lun 0
da5 at mps0 bus 0 scbus0 target 14 lun 0
da3 at mps0 bus 0 scbus0 target 12 lun 0
da7 at mps0 bus 0 scbus0 target 16 lun 0
da8 at mps0 bus 0 scbus0 target 17 lun 0
da6 at mps0 bus 0 scbus0 target 15 lun 0
da9 at mps0 bus 0 scbus0 target 18 lun 0
da10 at mps0 bus 0 scbus0 target 19 lun 0
ses0 at mps0 bus 0 scbus0 target 8 lun 0
ses0: <LSI CORP SAS2X28 0717> Fixed Enclosure Services SCSI-5 device
ses0: 600.000MB/s transfers
ses0: Command Queueing enabled
ses0: SCSI-3 SES Device

Jason

On Fri, Feb 17, 2012 at 1:24 AM, Desai, Kashyap <Kashyap.Desai at lsi.com<mailto:Kashyap.Desai at lsi.com>> wrote:
Jason,

Me also surprised when I see so many queries on <mps>.
Really wonderful to know that there are good amount of <mpslsi> driver use as well.

I was under impression that if you keep same FW and just change Driver, there should not be any difference in Target IDs assigned to Device connected behind that HBA.

Is this possible for you to keep everything unchanged and just change Driver version and see how things behaves. Please share "dmesg" logs of your both drivers.  BTW, did you tested "13.00.00.00-fbsd" and "11.00.00.00"  on same machine ?

FYI: We never observe this kind of issue in our lab.

` Kashyap

From: Jason Wolfe [mailto:nitroboost at gmail.com<mailto:nitroboost at gmail.com>]
Sent: Friday, February 17, 2012 1:31 PM
To: Desai, Kashyap
Cc: freebsd-scsi at freebsd.org<mailto:freebsd-scsi at freebsd.org>
Subject: Re: LSI2008 controller clobbers first disk with new LSI mps driver
Kashyap,

Ah a response from LSI, that's a pleasant surprise :)  Everything you've stated looks correct to me, the FreeBSD developed driver that has been replaced by the LSI driver has no issues with either firmware.  Your likely aware, but just to confirm, here is the history of the 3 various LSI drivers that have the issue on the 10.00.02.00 FW:

11.00.00.00 - binary driver I had received from you guys in mid  2011, mpslsi.ko, one for each 7.2-RELEASE and 8.2-RELEASE

11.255.03.00-fbsd - initial LSI driver committed to 8-STABLE on 2/2, r230922

13.00.00.00-fbsd - commited to 8-STABLE on 2/14, r231680

I have about 40 boxes with the 10.00.02.00 FW I've tested, so I'm fairly certain it's not bad hardware or a fluke.  You guys haven't seen anything like this in house?  I'd hate to hear I have to update the FW on these boxes as they are all quite a ways from me, though it seems there is some way to work around the behavior in the driver as the FreeBSD one does?  I have a few of these boxes out of service so I'm game to try some things out should that help.

Thank for the response,
Jason

On Thu, Feb 16, 2012 at 11:29 PM, Desai, Kashyap <Kashyap.Desai at lsi.com<mailto:Kashyap.Desai at lsi.com>> wrote:
Jason,

I have gone through your data provided in this thread. It is well understood because of your descriptive data.

So What I understood here is:

1. You tested with HBA Fw "07.00.00.00" and "10.00.02.00"

2. you have run your test on two different LSI BIOS versions.
Grabbed from below line.
MPT2BIOS-7.11.00.00 (2010.07.29) / PRODUCT REVISION 7.00.00.00
MPT2BIOS-7.19.00.00 (2011.05.16) / PRODUCT REVISION 10.00.02.00

Now I am able to see below three difference in your setup.

See FW version and check starting target id, all three has different way of assigning TargetIDs.
For first two case target id start with "8" but SES device assignment is different.
Last case target id start with "1"

mps0: Firmware: 07.00.00.00, Driver: 11.255.03.00-fbsd (OR) 13.00.00.00-fbsd
> > <SEAGATE ST91000640SS 0001> at scbus0 target 8 lun 0 (pass0,da0)
> > <SEAGATE ST91000640SS 0001> at scbus0 target 9 lun 0 (pass1,da1)
> > <SEAGATE ST91000640SS 0001> at scbus0 target 10 lun 0 (pass2,da2)
> > <SEAGATE ST91000640SS 0001> at scbus0 target 11 lun 0 (pass3,da3)
> > <SEAGATE ST91000640SS 0001> at scbus0 target 12 lun 0 (pass4,da4)
> > <SEAGATE ST91000640SS 0001> at scbus0 target 13 lun 0 (pass5,da5)
> > <SEAGATE ST91000640SS 0001> at scbus0 target 14 lun 0 (pass6,da6)
> > <SEAGATE ST91000640SS 0001> at scbus0 target 15 lun 0 (pass7,da7)
> > <SEAGATE ST91000640SS 0001> at scbus0 target 16 lun 0 (pass8,da8)
> > <SEAGATE ST91000640SS 0001> at scbus0 target 17 lun 0 (pass9,da9)
> > <SEAGATE ST91000640SS 0001> at scbus0 target 18 lun 0 (pass10,da10)
> > <SEAGATE ST91000640SS 0001> at scbus0 target 19 lun 0 (pass11,da11)
> > <LSI CORP SAS2X28 0717> at scbus0 target 20 lun 0 (ses0,pass12)

mps0: Firmware: 10.00.02.00, Driver: 13.00.00.00-fbsd
mps0: Firmware: 10.00.02.00, Driver: 11.00.00.00 (OR) 8.2-STABLE Inbox
> > <LSI CORP SAS2X28 0717> at scbus0 target 8 lun 0 (ses0,pass0)
> > <SEAGATE ST91000640SS 0001> at scbus0 target 9 lun 0 (da0,pass1)
> > <SEAGATE ST91000640SS 0001> at scbus0 target 10 lun 0 (da1,pass2)
> > <SEAGATE ST91000640SS 0001> at scbus0 target 11 lun 0 (da2,pass3)
> > <SEAGATE ST91000640SS 0001> at scbus0 target 12 lun 0 (da3,pass4)
> > <SEAGATE ST91000640SS 0001> at scbus0 target 13 lun 0 (da4,pass5)
> > <SEAGATE ST91000640SS 0001> at scbus0 target 14 lun 0 (da5,pass6)
> > <SEAGATE ST91000640SS 0001> at scbus0 target 15 lun 0 (da6,pass7)
> > <SEAGATE ST91000640SS 0001> at scbus0 target 16 lun 0 (da7,pass8)
> > <SEAGATE ST91000640SS 0001> at scbus0 target 17 lun 0 (da8,pass9)
> > <SEAGATE ST91000640SS 0001> at scbus0 target 18 lun 0 (da9,pass10)
> > <SEAGATE ST91000640SS 0001> at scbus0 target 19 lun 0 (da10,pass11)

On the FBSD developed driver active in 8-STABLE prior to the LSI
Release (Firmware: 10.00.02.00)

> > <SEAGATE ST91000640SS 0001> at scbus0 target 1 lun 0 (pass0,da0)
> > <SEAGATE ST91000640SS 0001> at scbus0 target 2 lun 0 (pass1,da1)
> > <SEAGATE ST91000640SS 0001> at scbus0 target 3 lun 0 (pass2,da2)
> > <SEAGATE ST91000640SS 0001> at scbus0 target 4 lun 0 (pass3,da3)
> > <SEAGATE ST91000640SS 0001> at scbus0 target 5 lun 0 (pass4,da4)
> > <SEAGATE ST91000640SS 0001> at scbus0 target 6 lun 0 (pass5,da5)
> > <SEAGATE ST91000640SS 0001> at scbus0 target 7 lun 0 (pass6,da6)
> > <SEAGATE ST91000640SS 0001> at scbus0 target 8 lun 0 (pass7,da7)
> > <SEAGATE ST91000640SS 0001> at scbus0 target 9 lun 0 (pass8,da8)
> > <SEAGATE ST91000640SS 0001> at scbus0 target 10 lun 0 (pass9,da9)
> > <SEAGATE ST91000640SS 0001> at scbus0 target 11 lun 0 (pass10,da10)
> > <SEAGATE ST91000640SS 0001> at scbus0 target 12 lun 0 (pass11,da11)
> > <LSI CORP SAS2X28 0717> at scbus0 target 13 lun 0 (ses0,pass12)

In summary,  (please confirm)
1. you have not seen any issue if you use "07.00.00.00" FW version.
2. _but_ when you use "10.00.02.00" FW, with "13.00.00.00-fbsd" driver version you are seeing
SES is detected before Drives as pass0.
3. When you use "10.00.02.00" FW with 8-STABLE inbox FBSD driver, you are finding SES device detected after Drives.

All driver is doing here is asking CAM layer to scan Bus when there is any device added on that bus.
So depending upon actual target Id  assigned by FW, it will be detected to camcontrol.

So bottom line is FW plays major role in sequencing Drives behind LSI controller.

 ~ Kashyap