Disk reordering on LSI SAS2008/mps(4)

[Jason Wolfe]

Hi!

We've been having an issue where we see some disk reordering on boot on 
HEAD from mid July on LSI controllers, maybe 5% of the time. We brought 
mps current as of r322364 with no change behavior.

I have a few logs attached with various debug output. In all cases I've 
seen the pass ordering to be proper, and cam does try to resolve the da 
ordering, but the device it tries to reassign to is already taken. 
Attached is the full output, and listing some relevant bits below for 
the casual reader. Being that the functionality in scsi_da.c has been 
fairly static, and it's attempting to reassign, it seems more likely we 
are running into something in mps here. The targets always look to be 
proper.

The various settings of hw.mps.use_phy_num (-1/0/1) don't change the 
behavior, and neither does hw.mps.enable_ssu=0. We have machines over 
various FW versions (15/16) that see the issue. I'm wondering if the 
fact that we see this issue over soft reboots means that the firmware 
isn't coming into play. To confirm, we are booting from the controller, 
so the LSI BIOS is enabled.

mps0 at pci0:3:0:0:        class=0x010700 card=0x040015d9 chip=0x00721000 
rev=0x03 hdr=0x00
     vendor     = 'LSI Logic / Symbios Logic'
     device     = 'SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon]'
     class      = mass storage
     subclass   = SAS

reorder-verbose.txt:
boot_verbose="YES"
hw.mps.debug_level="71"

da0 at mps0 bus 0 scbus0 target 17 lun 0
cam_periph_alloc: attempt to re-allocate valid device da0 rejected flags 
0x102 refcount 4
da1 at mps0 bus 0 scbus0 target 8 lun 0
daasync: Unable to attach to new device due to status 0x6
da2 at mps0 bus 0 scbus0 target 9 lun 0
...
da8 at mps0 bus 0 scbus0 target 15 lun 0
da9 at mps0 bus 0 scbus0 target 16 lun 0
da10 at mps0 bus 0 scbus0 target 18 lun 0
da11 at mps0 bus 0 scbus0 target 19 lun 0

pass0 at mps0 bus 0 scbus0 target 8 lun 0
pass1 at mps0 bus 0 scbus0 target 9 lun 0
...
pass9 at mps0 bus 0 scbus0 target 17 lun 0
pass10 at mps0 bus 0 scbus0 target 18 lun 0
pass11 at mps0 bus 0 scbus0 target 19 lun 0




reorder-mps-mapping.txt:
hw.mps.debug_level="583"

da0 at mps0 bus 0 scbus0 target 19 lun 0
da1 at mps0 bus 0 scbus0 target 8 lun 0
da2 at mps0 bus 0 scbus0 target 9 lun 0
...
da9 at mps0 bus 0 scbus0 target 16 lun 0
da10 at mps0 bus 0 scbus0 target 17 lun 0
da11 at mps0 bus 0 scbus0 target 18 lun 0
cam_periph_alloc: attempt to re-allocate valid device da0 rejected flags 
0x106 refcount 6
daasync: Unable to attach to new device due to status 0x6

ses0: da1,pass0: Element descriptor: 'Slot 01'
ses0: da1,pass0: SAS Device Slot Element: 1 Phys at Slot 0
ses0: da0,pass11: Element descriptor: 'Slot 12'
ses0: da0,pass11: SAS Device Slot Element: 1 Phys at Slot 11


Luckily we have found a way to fairly easily repro it over a few hours, 
so we are open to any suggestions.

Thanks!
Jason