mpr vs mps performance
Alan Somers
asomers at freebsd.org
Tue Oct 7 16:11:54 UTC 2014
On Tue, Oct 7, 2014 at 9:25 AM, Stephen Mcconnell
<stephen.mcconnell at avagotech.com> wrote:
>> -----Original Message-----
>> From: Stephen Mcconnell [mailto:stephen.mcconnell at avagotech.com]
>> Sent: Monday, October 06, 2014 4:55 PM
>> To: 'Alan Somers'; 'Borja Marcos'
>> Cc: 'FreeBSD-scsi'
>> Subject: RE: mpr vs mps performance
>>
>>
>>
>> > -----Original Message-----
>> > From: asomers at gmail.com [mailto:asomers at gmail.com] On Behalf Of Alan
>> > Somers
>> > Sent: Monday, October 06, 2014 10:13 AM
>> > To: Borja Marcos; slm at freebsd.org
>> > Cc: FreeBSD-scsi
>> > Subject: Re: mpr vs mps performance
>> >
>> > On Mon, Sep 22, 2014 at 9:39 AM, Borja Marcos <borjam at sarenet.es> wrote:
>> > >
>> > > Hello,
>> > >
>> > > I have been playing with the new SAS3 cards supported by the mpr
>> > > driver, and I´ve found out that they are, in the same hardware
>> > > configuration,
>> > considerably slower writing data. Moreover, running two simultaneous
>> "bonnie"
>> > benchmarks (I am using SSDs, and one "bonnie" sometimes hits 100% CPU
>> > usage, unable to really saturate the I/O) I see the writing activity
>> > somewhat stalling, with disk bandwidth going from 600 MB/s to around
>> > 50 for 20 seconds or so.
>> > >
>> > > I'd like to know if this matches anyone else's experiences. Also, I
>> > > can try and
>> > make some tests if needed. But for now it seems we will stick to the
>> > SAS2
>> HBAs.
>> > >
>> > >
>> > >
>> > >
>> > > The Bonnie results are:
>> > >
>> > > With mpr driver, SAS3:
>> > >
>> > > (each bonnie instance, so multiply the results by 2 to get the
>> > > actual bandwidth achieved)
>> > >
>> > > Seq output: (writing)
>> > > Block: 292155 KB/s
>> > > Rewrite: 139713 KB/s
>> > > Seq input:
>> > > Block: 862861 KB/s
>> > >
>> > > With mps driver: SAS2, again, total is 2x the following figures.
>> > >
>> > > Seq output: (writing)
>> > > Block: 587950 KB/s
>> > > Rewrite: 208239 KB/s
>> > > Seq. input: (reading)
>> > > Block: 842169 KB/s
>> > >
>> > >
>> > > The storage is a ZFS pool with a 9-disk raidz2 vdev, made of
>> > > Samsung
>> > > 840 EVO 1 TB SSDs. The pool has been created with an ashift of 12
>> > > (zpool applied it thanks to the 4 KB block quirk for these SSDs)
>> > >
>> > > <ATA Samsung SSD 840 BB0Q> at scbus0 target 9 lun 0
>> > > (pass0,da0)
>> > > <ATA Samsung SSD 840 BB0Q> at scbus0 target 10 lun 0
>> > > (pass1,da1)
>> > > <ATA Samsung SSD 840 BB0Q> at scbus0 target 11 lun 0
>> > > (pass2,da2)
>> > > <ATA Samsung SSD 840 BB0Q> at scbus0 target 12 lun 0
>> > > (pass3,da3)
>> > > <ATA Samsung SSD 840 BB0Q> at scbus0 target 13 lun 0
>> > > (pass4,da4)
>> > > <ATA Samsung SSD 840 BB0Q> at scbus0 target 14 lun 0
>> > > (pass5,da5)
>> > > <ATA Samsung SSD 840 BB0Q> at scbus0 target 17 lun 0
>> > > (pass7,da6)
>> > > <ATA Samsung SSD 840 BB0Q> at scbus0 target 18 lun 0
>> > > (pass8,da7)
>> > > <ATA Samsung SSD 840 BB0Q> at scbus0 target 27 lun 0
>> > > (pass14,da12)
>> > >
>> > >
>> > >
>> > >
>> > > The mpr card details follow:
>> > > Sep 17 09:49:39 elibm kernel: mpr0: <LSI SAS3008> port 0x3f00-0x3fff
>> > > mem
>> > 0x912f0000-0x912fffff irq 32 at device 0.0 on pci17
>> > > Sep 17 09:49:39 elibm kernel: mpr0: IOCFacts :
>> > > Sep 17 09:49:39 elibm kernel: MsgVersion: 0x205 Sep 17 09:49:39
>> > > elibm
>> > > kernel: HeaderVersion: 0x1d00 Sep 17 09:49:39 elibm kernel: IOCNumber:
>> > > 0 Sep 17 09:49:39 elibm kernel: IOCExceptions: 0x0 Sep 17 09:49:39
>> > > elibm kernel: MaxChainDepth: 128 Sep 17 09:49:39 elibm kernel:
>> > > NumberOfPorts: 1 Sep 17 09:49:39 elibm kernel: RequestCredit: 11264
>> > > Sep 17 09:49:39 elibm kernel: ProductID: 0x2221 Sep 17 09:49:39
>> > > elibm
>> > > kernel: IOCRequestFrameSize: 32 Sep 17 09:49:39 elibm kernel:
>> > > MaxInitiators: 1 Sep 17 09:49:39 elibm kernel: MaxTargets: 1024 Sep
>> > > 17
>> > > 09:49:39 elibm kernel: MaxSasExpanders: 14 Sep 17 09:49:39 elibm
>> > > kernel: MaxEnclosures: 15 Sep 17 09:49:39 elibm kernel:
>> > > HighPriorityCredit: 60 Sep 17 09:49:39 elibm kernel:
>> > > MaxReplyDescriptorPostQueueDepth: 65504 Sep 17 09:49:39 elibm kernel:
>> > > ReplyFrameSize: 32 Sep 17 09:49:39 elibm kernel: MaxVolumes: 0 Sep
>> > > 17
>> > > 09:49:39 elibm kernel: MaxDevHandle: 1047 Sep 17 09:49:39 elibm
>> > > kernel: MaxPersistentEntries: 128 Sep 17 09:49:39 elibm kernel: mpr0:
>> > > Firmware: 01.00.03.00, Driver: 05.255.05.00-fbsd Sep 17 09:49:39
>> > > elibm
>> > > kernel: mpr0: IOCCapabilities:
>> > > 3a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSI
>> > > XI
>> > > ndex,HostDisc>
>> > >
>> > >
>> > > And the mps card is a classic:
>> > > Sep 22 17:18:24 elibm kernel: mps0: <LSI SAS2008> port 0x3f00-0x3fff
>> > > mem 0x90ebc000-0x90ebffff,0x912c0000-0x912fffff irq 32 at device 0.0
>> > > on pci17 Sep 22 17:18:24 elibm kernel: mps0: Firmware: 18.00.00.00,
>> > > Driver: 19.00.00.00-fbsd Sep 22 17:18:24 elibm kernel: mps0:
>> > > IOCCapabilities:
>> > > 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,Hos
>> > > tD
>> > > isc>
>> > >
>> > >
>> > > The connected devices follow. Both use the same hardware (except for
>> > > the
>> > cables and HBA of course), but currently there's no way to check this
>> > with the
>> > SAS3 card, as sas3ircu nor sas3flash detect it on FreeBSD.
>> > >
>> > >
>> > >
>> > > # sas2ircu 0 display
>> > > LSI Corporation SAS2 IR Configuration Utility.
>> > > Version 18.00.00.00 (2013.11.18)
>> > > Copyright (c) 2009-2013 LSI Corporation. All rights reserved.
>> > >
>> > > Read configuration has been initiated for controller 0
>> > > --------------------------------------------------------------------
>> > > --
>> > > --
>> > > Controller information
>> > > ------------------------------------------------------------------------
>> > > Controller type : SAS2008
>> > > BIOS version : 7.35.00.00
>> > > Firmware version : 18.00.00.00
>> > > Channel description : 1 Serial Attached SCSI
>> > > Initiator ID : 0
>> > > Maximum physical devices : 255
>> > > Concurrent commands supported : 3432
>> > > Slot : 3
>> > > Segment : 0
>> > > Bus : 17
>> > > Device : 0
>> > > Function : 0
>> > > RAID Support : No
>> > > --------------------------------------------------------------------
>> > > --
>> > > --
>> > > IR Volume information
>> > > --------------------------------------------------------------------
>> > > --
>> > > --
>> > > --------------------------------------------------------------------
>> > > --
>> > > --
>> > > Physical device information
>> > > --------------------------------------------------------------------
>> > > --
>> > > --
>> > > Initiator at ID #0
>> > >
>> > > Device is a Hard disk
>> > > Enclosure # : 2
>> > > Slot # : 16
>> > > SAS Address : 5000c50-0-05b5-ce25
>> > > State : Ready (RDY)
>> > > Size (in MB)/(in sectors) : 140014/286749479
>> > > Manufacturer : SEAGATE
>> > > Model Number : ST9146803SS
>> > > Firmware Revision : FS03
>> > > Serial No : 3SD02W5L
>> > > GUID : N/A
>> > > Protocol : SAS
>> > > Drive Type : SAS_HDD
>> > >
>> > > Device is a Hard disk
>> > > Enclosure # : 2
>> > > Slot # : 17
>> > > SAS Address : 5005076-0-3e8e-81a2
>> > > State : Ready (RDY)
>> > > Size (in MB)/(in sectors) : 953869/1953525167
>> > > Manufacturer : ATA
>> > > Model Number : Samsung SSD 840
>> > > Firmware Revision : BB0Q
>> > > Serial No : S1D9NEADA08549F
>> > > GUID : N/A
>> > > Protocol : SATA
>> > > Drive Type : SATA_SSD
>> > >
>> > > Device is a Hard disk
>> > > Enclosure # : 2
>> > > Slot # : 18
>> > > SAS Address : 5005076-0-3e8e-81a3
>> > > State : Ready (RDY)
>> > > Size (in MB)/(in sectors) : 953869/1953525167
>> > > Manufacturer : ATA
>> > > Model Number : Samsung SSD 840
>> > > Firmware Revision : BB0Q
>> > > Serial No : S1D9NEADA08548T
>> > > GUID : N/A
>> > > Protocol : SATA
>> > > Drive Type : SATA_SSD
>> > >
>> > > Device is a Hard disk
>> > > Enclosure # : 2
>> > > Slot # : 19
>> > > SAS Address : 5005076-0-3e8e-81a4
>> > > State : Ready (RDY)
>> > > Size (in MB)/(in sectors) : 953869/1953525167
>> > > Manufacturer : ATA
>> > > Model Number : Samsung SSD 840
>> > > Firmware Revision : BB0Q
>> > > Serial No : S1D9NEADA08568E
>> > > GUID : N/A
>> > > Protocol : SATA
>> > > Drive Type : SATA_SSD
>> > >
>> > > Device is a Hard disk
>> > > Enclosure # : 2
>> > > Slot # : 20
>> > > SAS Address : 5005076-0-3e8e-81a5
>> > > State : Ready (RDY)
>> > > Size (in MB)/(in sectors) : 953869/1953525167
>> > > Manufacturer : ATA
>> > > Model Number : Samsung SSD 840
>> > > Firmware Revision : BB0Q
>> > > Serial No : S1D9NEADA08547X
>> > > GUID : N/A
>> > > Protocol : SATA
>> > > Drive Type : SATA_SSD
>> > >
>> > > Device is a Hard disk
>> > > Enclosure # : 2
>> > > Slot # : 21
>> > > SAS Address : 5005076-0-3e8e-81a6
>> > > State : Ready (RDY)
>> > > Size (in MB)/(in sectors) : 953869/1953525167
>> > > Manufacturer : ATA
>> > > Model Number : Samsung SSD 840
>> > > Firmware Revision : BB0Q
>> > > Serial No : S1D9NEADA08518Y
>> > > GUID : N/A
>> > > Protocol : SATA
>> > > Drive Type : SATA_SSD
>> > >
>> > > Device is a Hard disk
>> > > Enclosure # : 2
>> > > Slot # : 22
>> > > SAS Address : 5005076-0-3e8e-81a7
>> > > State : Ready (RDY)
>> > > Size (in MB)/(in sectors) : 953869/1953525167
>> > > Manufacturer : ATA
>> > > Model Number : Samsung SSD 840
>> > > Firmware Revision : BB0Q
>> > > Serial No : S1D9NEADA08556K
>> > > GUID : N/A
>> > > Protocol : SATA
>> > > Drive Type : SATA_SSD
>> > >
>> > > Device is a Enclosure services device
>> > > Enclosure # : 2
>> > > Slot # : 255
>> > > SAS Address : 5005076-0-3e8e-81b9
>> > > State : Standby (SBY)
>> > > Manufacturer : IBM-ESXS
>> > > Model Number : SAS EXP BP
>> > > Firmware Revision : 61A6
>> > > Serial No : 00000006
>> > > GUID : N/A
>> > > Protocol : SAS
>> > > Device Type : Enclosure services device
>> > >
>> > > Device is a Hard disk
>> > > Enclosure # : 3
>> > > Slot # : 0
>> > > SAS Address : 5005076-0-3e8e-86e9
>> > > State : Ready (RDY)
>> > > Size (in MB)/(in sectors) : 953869/1953525167
>> > > Manufacturer : ATA
>> > > Model Number : Samsung SSD 840
>> > > Firmware Revision : BB0Q
>> > > Serial No : S1D9NEADA08550R
>> > > GUID : N/A
>> > > Protocol : SATA
>> > > Drive Type : SATA_SSD
>> > >
>> > > Device is a Hard disk
>> > > Enclosure # : 3
>> > > Slot # : 1
>> > > SAS Address : 5005076-0-3e8e-86ea
>> > > State : Ready (RDY)
>> > > Size (in MB)/(in sectors) : 953869/1953525167
>> > > Manufacturer : ATA
>> > > Model Number : Samsung SSD 840
>> > > Firmware Revision : BB0Q
>> > > Serial No : S1D9NEADA08911Y
>> > > GUID : N/A
>> > > Protocol : SATA
>> > > Drive Type : SATA_SSD
>> > >
>> > > Device is a Hard disk
>> > > Enclosure # : 3
>> > > Slot # : 2
>> > > SAS Address : 5005076-0-3e8e-86eb
>> > > State : Ready (RDY)
>> > > Size (in MB)/(in sectors) : 953869/1953525167
>> > > Manufacturer : ATA
>> > > Model Number : Samsung SSD 840
>> > > Firmware Revision : BB0Q
>> > > Serial No : S1D9NEADA08811L
>> > > GUID : N/A
>> > > Protocol : SATA
>> > > Drive Type : SATA_SSD
>> > >
>> > > Device is a Hard disk
>> > > Enclosure # : 3
>> > > Slot # : 13
>> > > SAS Address : 5000c50-0-05b5-e531
>> > > State : Ready (RDY)
>> > > Size (in MB)/(in sectors) : 140014/286749479
>> > > Manufacturer : SEAGATE
>> > > Model Number : ST9146803SS
>> > > Firmware Revision : FS03
>> > > Serial No : 3SD02STR
>> > > GUID : N/A
>> > > Protocol : SAS
>> > > Drive Type : SAS_HDD
>> > >
>> > > Device is a Hard disk
>> > > Enclosure # : 3
>> > > Slot # : 14
>> > > SAS Address : 5000c50-0-05b5-d489
>> > > State : Ready (RDY)
>> > > Size (in MB)/(in sectors) : 140014/286749479
>> > > Manufacturer : SEAGATE
>> > > Model Number : ST9146803SS
>> > > Firmware Revision : FS03
>> > > Serial No : 3SD02TV1
>> > > GUID : N/A
>> > > Protocol : SAS
>> > > Drive Type : SAS_HDD
>> > >
>> > > Device is a Hard disk
>> > > Enclosure # : 3
>> > > Slot # : 15
>> > > SAS Address : 5000c50-0-05b5-f0ad
>> > > State : Ready (RDY)
>> > > Size (in MB)/(in sectors) : 140014/286749479
>> > > Manufacturer : SEAGATE
>> > > Model Number : ST9146803SS
>> > > Firmware Revision : FS03
>> > > Serial No : 3SD03F4C
>> > > GUID : N/A
>> > > Protocol : SAS
>> > > Drive Type : SAS_HDD
>> > >
>> > > Device is a Enclosure services device
>> > > Enclosure # : 3
>> > > Slot # : 255
>> > > SAS Address : 5005076-0-3e8e-86f9
>> > > State : Standby (SBY)
>> > > Manufacturer : IBM-ESXS
>> > > Model Number : SAS EXP BP
>> > > Firmware Revision : 61A6
>> > > Serial No : 00000006
>> > > GUID : N/A
>> > > Protocol : SAS
>> > > Device Type : Enclosure services device
>> > > --------------------------------------------------------------------
>> > > --
>> > > --
>> > > Enclosure information
>> > > ------------------------------------------------------------------------
>> > > Enclosure# : 1
>> > > Logical ID : 500605b0:07ba2100
>> > > Numslots : 8
>> > > StartSlot : 0
>> > > Enclosure# : 2
>> > > Logical ID : 50050760:3e8e81a0
>> > > Numslots : 25
>> > > StartSlot : 0
>> > > Enclosure# : 3
>> > > Logical ID : 50050760:3e8e86e0
>> > > Numslots : 25
>> > > StartSlot : 0
>> > > --------------------------------------------------------------------
>> > > --
>> > > --
>> > > SAS2IRCU: Command DISPLAY Completed Successfully.
>> > > SAS2IRCU: Utility Completed Successfully.
>> > >
>> > >
>> > >
>> > >
>> > > _______________________________________________
>> > > freebsd-scsi at freebsd.org mailing list
>> > > http://lists.freebsd.org/mailman/listinfo/freebsd-scsi
>> > > To unsubscribe, send any mail to
>> > > "freebsd-scsi-unsubscribe at freebsd.org"
>> >
>> > Adding Steve McConnell, who may not be on the freebsd-scsi list.
>>
>> Right, I'm not on the freebsd-scsi list. I suspect that the reason for
>> this is that
>> most of the Gen3 performance gains come from MSI-x support, but that is
>> not in
>> the mpr driver yet, and MSI-x is not supported in the 2008 card. But, I
>> have a
>> question out to another person and I'm waiting for more info.
>>
>> Steve
>
> Borja, can you upgrade your card to the latest FW. Your version is really
> old (1.00.03.00). We should be up to 7.00.00.00 now. There could be a lot
> of fixes that would affect performance.
To update the firmware, you must download LSI's
Installer_P6_for_FreeBSD from the link below. Then the tricky part:
you must hexedit the executable and replace all "mps3_" strings with
"mpr", suitably padded with NULLs. It sucks, I know; complain to
LSI/Avago. Then you must download the
9300_8e_Package_P6_IT_Firmware_BIOS_for_MSDOS_Windows (or whichever is
appropriate for your hardware) to get the actual firmware files. Then
you can use the hacked FreeBSD installer binary to download the
firmware from the Windows firmware package.
http://www.lsi.com/support/pages/download-results.aspx?component=Storage+Component&productfamily=Host+Bus+Adapters&productcode=P00766&assettype=0&productname=LSI+SAS+9300-8e
-Alan
More information about the freebsd-scsi
mailing list