Disk reordering on LSI SAS2008/mps(4)

[Kevin Bowling]

Note that we only see this bug with EARLY_AP_STARTUP enabled

Regards,

On Fri, Aug 25, 2017 at 2:11 PM, Jason Wolfe <j at nitrology.com> wrote:

> Attachments are useful.
>
>
> On 2017-08-25 13:58, Jason Wolfe wrote:
>
>> Hi!
>>
>> We've been having an issue where we see some disk reordering on boot
>> on HEAD from mid July on LSI controllers, maybe 5% of the time. We
>> brought mps current as of r322364 with no change behavior.
>>
>> I have a few logs attached with various debug output. In all cases
>> I've seen the pass ordering to be proper, and cam does try to resolve
>> the da ordering, but the device it tries to reassign to is already
>> taken. Attached is the full output, and listing some relevant bits
>> below for the casual reader. Being that the functionality in scsi_da.c
>> has been fairly static, and it's attempting to reassign, it seems more
>> likely we are running into something in mps here. The targets always
>> look to be proper.
>>
>> The various settings of hw.mps.use_phy_num (-1/0/1) don't change the
>> behavior, and neither does hw.mps.enable_ssu=0. We have machines over
>> various FW versions (15/16) that see the issue. I'm wondering if the
>> fact that we see this issue over soft reboots means that the firmware
>> isn't coming into play. To confirm, we are booting from the
>> controller, so the LSI BIOS is enabled.
>>
>> mps0 at pci0:3:0:0:        class=0x010700 card=0x040015d9 chip=0x00721000
>> rev=0x03 hdr=0x00
>>     vendor     = 'LSI Logic / Symbios Logic'
>>     device     = 'SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon]'
>>     class      = mass storage
>>     subclass   = SAS
>>
>> reorder-verbose.txt:
>> boot_verbose="YES"
>> hw.mps.debug_level="71"
>>
>> da0 at mps0 bus 0 scbus0 target 17 lun 0
>> cam_periph_alloc: attempt to re-allocate valid device da0 rejected
>> flags 0x102 refcount 4
>> da1 at mps0 bus 0 scbus0 target 8 lun 0
>> daasync: Unable to attach to new device due to status 0x6
>> da2 at mps0 bus 0 scbus0 target 9 lun 0
>> ...
>> da8 at mps0 bus 0 scbus0 target 15 lun 0
>> da9 at mps0 bus 0 scbus0 target 16 lun 0
>> da10 at mps0 bus 0 scbus0 target 18 lun 0
>> da11 at mps0 bus 0 scbus0 target 19 lun 0
>>
>> pass0 at mps0 bus 0 scbus0 target 8 lun 0
>> pass1 at mps0 bus 0 scbus0 target 9 lun 0
>> ...
>> pass9 at mps0 bus 0 scbus0 target 17 lun 0
>> pass10 at mps0 bus 0 scbus0 target 18 lun 0
>> pass11 at mps0 bus 0 scbus0 target 19 lun 0
>>
>>
>>
>>
>> reorder-mps-mapping.txt:
>> hw.mps.debug_level="583"
>>
>> da0 at mps0 bus 0 scbus0 target 19 lun 0
>> da1 at mps0 bus 0 scbus0 target 8 lun 0
>> da2 at mps0 bus 0 scbus0 target 9 lun 0
>> ...
>> da9 at mps0 bus 0 scbus0 target 16 lun 0
>> da10 at mps0 bus 0 scbus0 target 17 lun 0
>> da11 at mps0 bus 0 scbus0 target 18 lun 0
>> cam_periph_alloc: attempt to re-allocate valid device da0 rejected
>> flags 0x106 refcount 6
>> daasync: Unable to attach to new device due to status 0x6
>>
>> ses0: da1,pass0: Element descriptor: 'Slot 01'
>> ses0: da1,pass0: SAS Device Slot Element: 1 Phys at Slot 0
>> ses0: da0,pass11: Element descriptor: 'Slot 12'
>> ses0: da0,pass11: SAS Device Slot Element: 1 Phys at Slot 11
>>
>>
>> Luckily we have found a way to fairly easily repro it over a few
>> hours, so we are open to any suggestions.
>>
>> Thanks!
>> Jason
>
>
> _______________________________________________
> freebsd-scsi at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-scsi
> To unsubscribe, send any mail to "freebsd-scsi-unsubscribe at freebsd.org"
>