LSI2008 controller clobbers first disk with new LSI mps driver

Fri Feb 17 17:16:23 UTC 2012

On Fri, Feb 17, 2012 at 1:07 AM, Desai, Kashyap <Kashyap.Desai at lsi.com> wrote:
> Jason,
>
> Can you switch your mail client to "text mode". It is difficult to do inline reply in html format.
>
> <mpslsi> version 08.55.00.00, <mps> version 11.255.03.00-fbsd and <mps> version 13.00.00.00-fbsd detected SES device as Target Id = 8.
> This target Id is retrieved from FW. I am able to map my theory in your case.
>
> Now, Only problem is when you use "Old mps FreeBSD developers" things are different (and you would like to see same result using other driver versions ).
> Here is a reason why behavior is different.
> Our older mps driver does not had mapping code available in few driver releases. (see mps_mapping.c file)
> Because of that reason Driver will assign Target Id sequential (Internal sequence) from Event received from FW.
>
> Summary is all driver behavior with FW 10.00.02.00 is as expected.
>
> Can you boot into verbose mode and send me log where it actual hangs ?  Also what is your /etc/fstab entry.

we've actually had exactly the same problem with 16-port LSI card
running IT (non-RAID) firmware. i was looking at the source and found
that driver does some sort of persistent mapping for scsi target.
personally, i do net get why its required, especially when running in
IT mode.

to fix this issue we had to "lobotomize" LSI mps(4) card. the
following procedure is know to work (i'm typing from memory, so double
check)

1) start lsiutil

2) select card

3) enable expert mode

4) select option 33 then sub option 7 (erase all non-production
configuration or something like that)

5) return to main menu and select option 61 (reset to all defaults)

6) select option 37 (force full rescan)

repeat steps 2 - 6 for every card and reboot. activate LSI
configuration bios utility (ctrl+c) and do optional post-configuration
(we disable boot order on LSI adapters). save and reboot.

we also use hints system to ensure drives/ports stay wired correctly
to their respective device nodes.

hope this helps.

thanks,
max

> From: Jason Wolfe [mailto:nitroboost at gmail.com]
> Sent: Friday, February 17, 2012 2:19 PM
> To: Desai, Kashyap
> Cc: freebsd-scsi at freebsd.org
> Subject: Re: LSI2008 controller clobbers first disk with new LSI mps driver
>
> Kashyap,
>
> Yes I've run all 4 drivers on the same machine with the 10.00.02.00 FW:
>
> Old mps from (I believe) FreeBSD developers:
>
> mps0: <LSI SAS2008> port 0xe000-0xe0ff mem 0xfbd3c000-0xfbd3ffff,0xfbd40000-0xfbd7ffff irq 26 at device 0.0 on pci4
> mps0: Firmware: 10.00.02.00
> mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
> mps0: [ITHREAD]
> da0 at mps0 bus 0 scbus0 target 1 lun 0
> da1 at mps0 bus 0 scbus0 target 2 lun 0
> da2 at mps0 bus 0 scbus0 target 3 lun 0
> da3 at mps0 bus 0 scbus0 target 4 lun 0
> da4 at mps0 bus 0 scbus0 target 5 lun 0
> da5 at mps0 bus 0 scbus0 target 6 lun 0
> da6 at mps0 bus 0 scbus0 target 7 lun 0
> da8 at mps0 bus 0 scbus0 target 9 lun 0
> da7 at mps0 bus 0 scbus0 target 8 lun 0
> da10 at mps0 bus 0 scbus0 target 11 lun 0
> da9 at mps0 bus 0 scbus0 target 10 lun 0
> da11 at mps0 bus 0 scbus0 target 12 lun 0
> ses0 at mps0 bus 0 scbus0 target 13 lun 0
> ses0: <LSI CORP SAS2X28 0717> Fixed Enclosure Services SCSI-5 device
> ses0: 600.000MB/s transfers
> ses0: Command Queueing enabled
> ses0: SCSI-3 SES Device
>
>
>
> mpslsi.ko binary from 2011:
>
> mpslsi0: <LSI SAS2008> port 0xe000-0xe0ff mem 0xfbd3c000-0xfbd3ffff,0xfbd40000-0xfbd7ffff irq 26 at device 0.0 on pci4
> mpslsi0: Firmware: 10.00.02.00, Driver: 08.55.00.00
> mpslsi0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
> mpslsi0: [ITHREAD]
> ses0 at mpslsi0 bus 0 target 8 lun 0
> ses0: <LSI CORP SAS2X28 0717> Fixed Enclosure Services SCSI-5 device
> ses0: 600.000MB/s transfers
> ses0: Command Queueing Enabled
> ses0: SCSI-3 SES Device
> da0 at mpslsi0 bus 0 target 9 lun 0
> da1 at mpslsi0 bus 0 target 10 lun 0
> da2 at mpslsi0 bus 0 target 11 lun 0
> da3 at mpslsi0 bus 0 target 12 lun 0
> da4 at mpslsi0 bus 0 target 13 lun 0
> da5 at mpslsi0 bus 0 target 14 lun 0
> da6 at mpslsi0 bus 0 target 15 lun 0
> da7 at mpslsi0 bus 0 target 16 lun 0
> da8 at mpslsi0 bus 0 target 17 lun 0
> da9 at mpslsi0 bus 0 target 18 lun 0
> da10 at mpslsi0 bus 0 target 19 lun 0
>
>
>
> 11.255.03.00-fbsd:
>
> mps0: <LSI SAS2008> port 0xe000-0xe0ff mem 0xfbd3c000-0xfbd3ffff,0xfbd40000-0xfbd7ffff irq 26 at device 0.0 on pci4
> mps0: Firmware: 10.00.02.00, Driver: 11.255.03.00-fbsd
> mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
> mps0: [ITHREAD]
> ses0 at mps0 bus 0 scbus0 target 8 lun 0
> ses0: <LSI CORP SAS2X28 0717> Fixed Enclosure Services SCSI-5 device
> ses0: 600.000MB/s transfers
> ses0: Command Queueing enabled
> ses0: SCSI-3 SES Device
> da0 at mps0 bus 0 scbus0 target 9 lun 0
> da1 at mps0 bus 0 scbus0 target 10 lun 0
> da2 at mps0 bus 0 scbus0 target 11 lun 0
> da3 at mps0 bus 0 scbus0 target 12 lun 0
> da4 at mps0 bus 0 scbus0 target 13 lun 0
> da5 at mps0 bus 0 scbus0 target 14 lun 0
> da6 at mps0 bus 0 scbus0 target 15 lun 0
> da7 at mps0 bus 0 scbus0 target 16 lun 0
> da8 at mps0 bus 0 scbus0 target 17 lun 0
> da9 at mps0 bus 0 scbus0 target 18 lun 0
> da10 at mps0 bus 0 scbus0 target 19 lun 0
>
>
>
> Current 13.00.00.00-fbsd:
>
> mps0: <LSI SAS2008> port 0xe000-0xe0ff mem 0xfbd3c000-0xfbd3ffff,0xfbd40000-0xfbd7ffff irq 26 at device 0.0 on pci4
> mps0: Firmware: 10.00.02.00, Driver: 13.00.00.00-fbsd
> mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
> mps0: [ITHREAD]
> da1 at mps0 bus 0 scbus0 target 10 lun 0
> da0 at mps0 bus 0 scbus0 target 9 lun 0
> da2 at mps0 bus 0 scbus0 target 11 lun 0
> da4 at mps0 bus 0 scbus0 target 13 lun 0
> da5 at mps0 bus 0 scbus0 target 14 lun 0
> da3 at mps0 bus 0 scbus0 target 12 lun 0
> da7 at mps0 bus 0 scbus0 target 16 lun 0
> da8 at mps0 bus 0 scbus0 target 17 lun 0
> da6 at mps0 bus 0 scbus0 target 15 lun 0
> da9 at mps0 bus 0 scbus0 target 18 lun 0
> da10 at mps0 bus 0 scbus0 target 19 lun 0
> ses0 at mps0 bus 0 scbus0 target 8 lun 0
> ses0: <LSI CORP SAS2X28 0717> Fixed Enclosure Services SCSI-5 device
> ses0: 600.000MB/s transfers
> ses0: Command Queueing enabled
> ses0: SCSI-3 SES Device
>
> Jason
>
> On Fri, Feb 17, 2012 at 1:24 AM, Desai, Kashyap <Kashyap.Desai at lsi.com<mailto:Kashyap.Desai at lsi.com>> wrote:
> Jason,
>
> Me also surprised when I see so many queries on <mps>.
> Really wonderful to know that there are good amount of <mpslsi> driver use as well.
>
> I was under impression that if you keep same FW and just change Driver, there should not be any difference in Target IDs assigned to Device connected behind that HBA.
>
> Is this possible for you to keep everything unchanged and just change Driver version and see how things behaves. Please share "dmesg" logs of your both drivers.  BTW, did you tested "13.00.00.00-fbsd" and "11.00.00.00"  on same machine ?
>
> FYI: We never observe this kind of issue in our lab.
>
> ` Kashyap
>
> From: Jason Wolfe [mailto:nitroboost at gmail.com<mailto:nitroboost at gmail.com>]
> Sent: Friday, February 17, 2012 1:31 PM
> To: Desai, Kashyap
> Cc: freebsd-scsi at freebsd.org<mailto:freebsd-scsi at freebsd.org>
> Subject: Re: LSI2008 controller clobbers first disk with new LSI mps driver
> Kashyap,
>
> Ah a response from LSI, that's a pleasant surprise :)  Everything you've stated looks correct to me, the FreeBSD developed driver that has been replaced by the LSI driver has no issues with either firmware.  Your likely aware, but just to confirm, here is the history of the 3 various LSI drivers that have the issue on the 10.00.02.00 FW:
>
> 11.00.00.00 - binary driver I had received from you guys in mid  2011, mpslsi.ko, one for each 7.2-RELEASE and 8.2-RELEASE
>
> 11.255.03.00-fbsd - initial LSI driver committed to 8-STABLE on 2/2, r230922
>
> 13.00.00.00-fbsd - commited to 8-STABLE on 2/14, r231680
>
> I have about 40 boxes with the 10.00.02.00 FW I've tested, so I'm fairly certain it's not bad hardware or a fluke.  You guys haven't seen anything like this in house?  I'd hate to hear I have to update the FW on these boxes as they are all quite a ways from me, though it seems there is some way to work around the behavior in the driver as the FreeBSD one does?  I have a few of these boxes out of service so I'm game to try some things out should that help.
>
> Thank for the response,
> Jason
>
> On Thu, Feb 16, 2012 at 11:29 PM, Desai, Kashyap <Kashyap.Desai at lsi.com<mailto:Kashyap.Desai at lsi.com>> wrote:
> Jason,
>
> I have gone through your data provided in this thread. It is well understood because of your descriptive data.
>
> So What I understood here is:
>
> 1. You tested with HBA Fw "07.00.00.00" and "10.00.02.00"
>
> 2. you have run your test on two different LSI BIOS versions.
> Grabbed from below line.
> MPT2BIOS-7.11.00.00 (2010.07.29) / PRODUCT REVISION 7.00.00.00
> MPT2BIOS-7.19.00.00 (2011.05.16) / PRODUCT REVISION 10.00.02.00
>
> Now I am able to see below three difference in your setup.
>
> See FW version and check starting target id, all three has different way of assigning TargetIDs.
> For first two case target id start with "8" but SES device assignment is different.
> Last case target id start with "1"
>
>
> mps0: Firmware: 07.00.00.00, Driver: 11.255.03.00-fbsd (OR) 13.00.00.00-fbsd
>> > <SEAGATE ST91000640SS 0001> at scbus0 target 8 lun 0 (pass0,da0)
>> > <SEAGATE ST91000640SS 0001> at scbus0 target 9 lun 0 (pass1,da1)
>> > <SEAGATE ST91000640SS 0001> at scbus0 target 10 lun 0 (pass2,da2)
>> > <SEAGATE ST91000640SS 0001> at scbus0 target 11 lun 0 (pass3,da3)
>> > <SEAGATE ST91000640SS 0001> at scbus0 target 12 lun 0 (pass4,da4)
>> > <SEAGATE ST91000640SS 0001> at scbus0 target 13 lun 0 (pass5,da5)
>> > <SEAGATE ST91000640SS 0001> at scbus0 target 14 lun 0 (pass6,da6)
>> > <SEAGATE ST91000640SS 0001> at scbus0 target 15 lun 0 (pass7,da7)
>> > <SEAGATE ST91000640SS 0001> at scbus0 target 16 lun 0 (pass8,da8)
>> > <SEAGATE ST91000640SS 0001> at scbus0 target 17 lun 0 (pass9,da9)
>> > <SEAGATE ST91000640SS 0001> at scbus0 target 18 lun 0 (pass10,da10)
>> > <SEAGATE ST91000640SS 0001> at scbus0 target 19 lun 0 (pass11,da11)
>> > <LSI CORP SAS2X28 0717> at scbus0 target 20 lun 0 (ses0,pass12)
>
> mps0: Firmware: 10.00.02.00, Driver: 13.00.00.00-fbsd
> mps0: Firmware: 10.00.02.00, Driver: 11.00.00.00 (OR) 8.2-STABLE Inbox
>> > <LSI CORP SAS2X28 0717> at scbus0 target 8 lun 0 (ses0,pass0)
>> > <SEAGATE ST91000640SS 0001> at scbus0 target 9 lun 0 (da0,pass1)
>> > <SEAGATE ST91000640SS 0001> at scbus0 target 10 lun 0 (da1,pass2)
>> > <SEAGATE ST91000640SS 0001> at scbus0 target 11 lun 0 (da2,pass3)
>> > <SEAGATE ST91000640SS 0001> at scbus0 target 12 lun 0 (da3,pass4)
>> > <SEAGATE ST91000640SS 0001> at scbus0 target 13 lun 0 (da4,pass5)
>> > <SEAGATE ST91000640SS 0001> at scbus0 target 14 lun 0 (da5,pass6)
>> > <SEAGATE ST91000640SS 0001> at scbus0 target 15 lun 0 (da6,pass7)
>> > <SEAGATE ST91000640SS 0001> at scbus0 target 16 lun 0 (da7,pass8)
>> > <SEAGATE ST91000640SS 0001> at scbus0 target 17 lun 0 (da8,pass9)
>> > <SEAGATE ST91000640SS 0001> at scbus0 target 18 lun 0 (da9,pass10)
>> > <SEAGATE ST91000640SS 0001> at scbus0 target 19 lun 0 (da10,pass11)
>
> On the FBSD developed driver active in 8-STABLE prior to the LSI
> Release (Firmware: 10.00.02.00)
>
>> > <SEAGATE ST91000640SS 0001> at scbus0 target 1 lun 0 (pass0,da0)
>> > <SEAGATE ST91000640SS 0001> at scbus0 target 2 lun 0 (pass1,da1)
>> > <SEAGATE ST91000640SS 0001> at scbus0 target 3 lun 0 (pass2,da2)
>> > <SEAGATE ST91000640SS 0001> at scbus0 target 4 lun 0 (pass3,da3)
>> > <SEAGATE ST91000640SS 0001> at scbus0 target 5 lun 0 (pass4,da4)
>> > <SEAGATE ST91000640SS 0001> at scbus0 target 6 lun 0 (pass5,da5)
>> > <SEAGATE ST91000640SS 0001> at scbus0 target 7 lun 0 (pass6,da6)
>> > <SEAGATE ST91000640SS 0001> at scbus0 target 8 lun 0 (pass7,da7)
>> > <SEAGATE ST91000640SS 0001> at scbus0 target 9 lun 0 (pass8,da8)
>> > <SEAGATE ST91000640SS 0001> at scbus0 target 10 lun 0 (pass9,da9)
>> > <SEAGATE ST91000640SS 0001> at scbus0 target 11 lun 0 (pass10,da10)
>> > <SEAGATE ST91000640SS 0001> at scbus0 target 12 lun 0 (pass11,da11)
>> > <LSI CORP SAS2X28 0717> at scbus0 target 13 lun 0 (ses0,pass12)
>
>
> In summary,  (please confirm)
> 1. you have not seen any issue if you use "07.00.00.00" FW version.
> 2. _but_ when you use "10.00.02.00" FW, with "13.00.00.00-fbsd" driver version you are seeing
> SES is detected before Drives as pass0.
> 3. When you use "10.00.02.00" FW with 8-STABLE inbox FBSD driver, you are finding SES device detected after Drives.
>
>
> All driver is doing here is asking CAM layer to scan Bus when there is any device added on that bus.
> So depending upon actual target Id  assigned by FW, it will be detected to camcontrol.
>
> So bottom line is FW plays major role in sequencing Drives behind LSI controller.
>
>  ~ Kashyap
>
> _______________________________________________
> freebsd-scsi at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-scsi
> To unsubscribe, send any mail to "freebsd-scsi-unsubscribe at freebsd.org"