LSI 9300-8e performance experiences / pcie negotiation / Down reving protocol

[John]

Hi Folks,

   Does anyone have any experience with the LSI 9300 HBA adapter? I've
been putting together a new server with a pair of these cards going
into a quad-cabled SuperMicro storage enclosure and am not seeing the
performance I think I should.

   I currently have 1 cable from each card going to each expander set. I
am unable to get better than 5.5GB/sec through-put.

   pciconf shows the following for the cards:

mpr0 at pci0:2:0:0:        class=0x010700 card=0x30a01000 chip=0x00971000 rev=0x02 hdr=0x00
    vendor     = 'LSI Logic / Symbios Logic'
    device     = 'SAS3008 PCI-Express Fusion-MPT SAS-3'
    class      = mass storage
    subclass   = SAS
    bar   [10] = type I/O Port, range 32, base 0x5000, size 256, enabled
    bar   [14] = type Memory, range 64, base 0xc7500000, size 65536, enabled
    cap 01[50] = powerspec 3  supports D0 D1 D2 D3  current D0
    cap 10[68] = PCI-Express 2 endpoint max data 256(4096) FLR link x8(x8)
                 speed 8.0(8.0)
    cap 03[d0] = VPD
    cap 05[a8] = MSI supports 1 message, 64 bit, vector masks 
    cap 11[c0] = MSI-X supports 96 messages, enabled
                 Table in map 0x14[0xe000], PBA in map 0x14[0xf000]
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected
    ecap 0019[1e0] = PCIe Sec 1 lane errors 0
    ecap 0004[1c0] = Power Budgeting 1
    ecap 0016[190] = DPA 1
    ecap 0010[150] = SRIOV 1
    ecap 000e[148] = ARI 1

   mpr1 is the same. I am curious about cap 10[68]. While the link widths
and speeds show correctly, it is a pcie3 card.  All firmware is up-to-date
(I've actually tried all of them from 1 to 6 (no P3)). Cards are plugged
into PCI-Express 3 x16 slots. From dmidecode --type 9:

	Designation: CPU1 SLOT3 PCI-E 3.0 X16
	Type: x16 PCI Express 3 x16


   3 x shelves: 24 drives ea. 72 total (da0 - da143)

# camcontrol inquiry ses1
pass49: <LSI SAS3x40 0501> Fixed Enclosure Services SCSI-5 device 
pass49: Serial Number         
pass49: 1200.000MB/s transfers, Command Queueing Enabled
# camcontrol inquiry da0
pass0: <SEAGATE ST1200MM0017 0002> Fixed Direct Access SCSI-6 device 
pass0: Serial Number S3L0EENZ0000M446AG8K
pass0: 1200.000MB/s transfers, Command Queueing Enabled

   gmultipath enabled
   alternating i/o paths.
   

   Booting the system with verbose enabled, I do see the following:

(probe0:mpr1:0:8:0): Down reving Protocol Version from 4 to 0?

   It looks like 1 of these messages for every disk, coming from
cam/scsi/scsi_xpt.c:

        if (cts->protocol_version > device->protocol_version) {
                if (bootverbose) {
                        xpt_print(path, "Down reving Protocol "
                            "Version from %d to %d?\n", cts->protocol_version,
                            device->protocol_version);
                }
                cts->protocol_version = device->protocol_version;
        }

   

   If anyone has any thoughts please let me know.

Thanks,
John

Current test:

Write test files:

   iozone -w -+N -+u -P 0 -i 0 -+n -s 16g -r 128k -t 24 -I -F /poollsi/01.tmp ...

export/import/rotate pool

Read test:

   iozone    -+N -+u -P 0 -i 1 -+n -s 16g -r 128k -t 24 -I -F /poollsi/01.tmp ...

Children see throughput for 24 readers 		= 5481194.48 KB/sec
	Parent sees throughput for 24 readers 		= 5478389.11 KB/sec
	Min throughput per process 			=  222687.09 KB/sec 
	Max throughput per process 			=  234463.22 KB/sec
	Avg throughput per process 			=  228383.10 KB/sec
	Min xfer 					= 15934592.00 KB
	CPU utilization: Wall time   71.607    CPU time  563.762    CPU utilization 787.30 % (out of 24 cpus)


IOCFacts:

mpr1: <LSI SAS3008> port 0xe000-0xe0ff mem 0xfbd00000-0xfbd0ffff irq 56 at device 0.0 on pci135
mpr1: IOCFacts  :
        MsgVersion: 0x205
        HeaderVersion: 0x2300
        IOCNumber: 0
        IOCExceptions: 0x0
        MaxChainDepth: 128
        NumberOfPorts: 1
        RequestCredit: 10176
        ProductID: 0x2221
        IOCRequestFrameSize: 32
        MaxInitiators: 32
        MaxTargets: 1024
        MaxSasExpanders: 64
        MaxEnclosures: 65
        HighPriorityCredit: 124
        MaxReplyDescriptorPostQueueDepth: 65504
        ReplyFrameSize: 32
        MaxVolumes: 0
        MaxDevHandle: 1128
        MaxPersistentEntries: 128
mpr1: Firmware: 06.00.00.00, Driver: 05.255.05.00-fbsd
mpr1: IOCCapabilities: 7a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc>
mpr1: attempting to allocate 1 MSI-X vectors (96 supported)
msi: routing MSI-X IRQ 324 to local APIC 0 vector 121
mpr1: using IRQ 324 for MSI-X