mpr vs mps performance
Stephen Mcconnell
stephen.mcconnell at avagotech.com
Tue Oct 7 15:33:36 UTC 2014
> -----Original Message-----
> From: Stephen Mcconnell [mailto:stephen.mcconnell at avagotech.com]
> Sent: Monday, October 06, 2014 4:55 PM
> To: 'Alan Somers'; 'Borja Marcos'
> Cc: 'FreeBSD-scsi'
> Subject: RE: mpr vs mps performance
>
>
>
> > -----Original Message-----
> > From: asomers at gmail.com [mailto:asomers at gmail.com] On Behalf Of Alan
> > Somers
> > Sent: Monday, October 06, 2014 10:13 AM
> > To: Borja Marcos; slm at freebsd.org
> > Cc: FreeBSD-scsi
> > Subject: Re: mpr vs mps performance
> >
> > On Mon, Sep 22, 2014 at 9:39 AM, Borja Marcos <borjam at sarenet.es> wrote:
> > >
> > > Hello,
> > >
> > > I have been playing with the new SAS3 cards supported by the mpr
> > > driver, and I´ve found out that they are, in the same hardware
> > > configuration,
> > considerably slower writing data. Moreover, running two simultaneous
> "bonnie"
> > benchmarks (I am using SSDs, and one "bonnie" sometimes hits 100% CPU
> > usage, unable to really saturate the I/O) I see the writing activity
> > somewhat stalling, with disk bandwidth going from 600 MB/s to around
> > 50 for 20 seconds or so.
> > >
> > > I'd like to know if this matches anyone else's experiences. Also, I
> > > can try and
> > make some tests if needed. But for now it seems we will stick to the
> > SAS2
> HBAs.
> > >
> > >
> > >
> > >
> > > The Bonnie results are:
> > >
> > > With mpr driver, SAS3:
> > >
> > > (each bonnie instance, so multiply the results by 2 to get the
> > > actual bandwidth achieved)
> > >
> > > Seq output: (writing)
> > > Block: 292155 KB/s
> > > Rewrite: 139713 KB/s
> > > Seq input:
> > > Block: 862861 KB/s
> > >
> > > With mps driver: SAS2, again, total is 2x the following figures.
> > >
> > > Seq output: (writing)
> > > Block: 587950 KB/s
> > > Rewrite: 208239 KB/s
> > > Seq. input: (reading)
> > > Block: 842169 KB/s
> > >
> > >
> > > The storage is a ZFS pool with a 9-disk raidz2 vdev, made of
> > > Samsung
> > > 840 EVO 1 TB SSDs. The pool has been created with an ashift of 12
> > > (zpool applied it thanks to the 4 KB block quirk for these SSDs)
> > >
> > > <ATA Samsung SSD 840 BB0Q> at scbus0 target 9 lun 0
> > > (pass0,da0)
> > > <ATA Samsung SSD 840 BB0Q> at scbus0 target 10 lun 0
> > > (pass1,da1)
> > > <ATA Samsung SSD 840 BB0Q> at scbus0 target 11 lun 0
> > > (pass2,da2)
> > > <ATA Samsung SSD 840 BB0Q> at scbus0 target 12 lun 0
> > > (pass3,da3)
> > > <ATA Samsung SSD 840 BB0Q> at scbus0 target 13 lun 0
> > > (pass4,da4)
> > > <ATA Samsung SSD 840 BB0Q> at scbus0 target 14 lun 0
> > > (pass5,da5)
> > > <ATA Samsung SSD 840 BB0Q> at scbus0 target 17 lun 0
> > > (pass7,da6)
> > > <ATA Samsung SSD 840 BB0Q> at scbus0 target 18 lun 0
> > > (pass8,da7)
> > > <ATA Samsung SSD 840 BB0Q> at scbus0 target 27 lun 0
> > > (pass14,da12)
> > >
> > >
> > >
> > >
> > > The mpr card details follow:
> > > Sep 17 09:49:39 elibm kernel: mpr0: <LSI SAS3008> port 0x3f00-0x3fff
> > > mem
> > 0x912f0000-0x912fffff irq 32 at device 0.0 on pci17
> > > Sep 17 09:49:39 elibm kernel: mpr0: IOCFacts :
> > > Sep 17 09:49:39 elibm kernel: MsgVersion: 0x205 Sep 17 09:49:39
> > > elibm
> > > kernel: HeaderVersion: 0x1d00 Sep 17 09:49:39 elibm kernel: IOCNumber:
> > > 0 Sep 17 09:49:39 elibm kernel: IOCExceptions: 0x0 Sep 17 09:49:39
> > > elibm kernel: MaxChainDepth: 128 Sep 17 09:49:39 elibm kernel:
> > > NumberOfPorts: 1 Sep 17 09:49:39 elibm kernel: RequestCredit: 11264
> > > Sep 17 09:49:39 elibm kernel: ProductID: 0x2221 Sep 17 09:49:39
> > > elibm
> > > kernel: IOCRequestFrameSize: 32 Sep 17 09:49:39 elibm kernel:
> > > MaxInitiators: 1 Sep 17 09:49:39 elibm kernel: MaxTargets: 1024 Sep
> > > 17
> > > 09:49:39 elibm kernel: MaxSasExpanders: 14 Sep 17 09:49:39 elibm
> > > kernel: MaxEnclosures: 15 Sep 17 09:49:39 elibm kernel:
> > > HighPriorityCredit: 60 Sep 17 09:49:39 elibm kernel:
> > > MaxReplyDescriptorPostQueueDepth: 65504 Sep 17 09:49:39 elibm kernel:
> > > ReplyFrameSize: 32 Sep 17 09:49:39 elibm kernel: MaxVolumes: 0 Sep
> > > 17
> > > 09:49:39 elibm kernel: MaxDevHandle: 1047 Sep 17 09:49:39 elibm
> > > kernel: MaxPersistentEntries: 128 Sep 17 09:49:39 elibm kernel: mpr0:
> > > Firmware: 01.00.03.00, Driver: 05.255.05.00-fbsd Sep 17 09:49:39
> > > elibm
> > > kernel: mpr0: IOCCapabilities:
> > > 3a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSI
> > > XI
> > > ndex,HostDisc>
> > >
> > >
> > > And the mps card is a classic:
> > > Sep 22 17:18:24 elibm kernel: mps0: <LSI SAS2008> port 0x3f00-0x3fff
> > > mem 0x90ebc000-0x90ebffff,0x912c0000-0x912fffff irq 32 at device 0.0
> > > on pci17 Sep 22 17:18:24 elibm kernel: mps0: Firmware: 18.00.00.00,
> > > Driver: 19.00.00.00-fbsd Sep 22 17:18:24 elibm kernel: mps0:
> > > IOCCapabilities:
> > > 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,Hos
> > > tD
> > > isc>
> > >
> > >
> > > The connected devices follow. Both use the same hardware (except for
> > > the
> > cables and HBA of course), but currently there's no way to check this
> > with the
> > SAS3 card, as sas3ircu nor sas3flash detect it on FreeBSD.
> > >
> > >
> > >
> > > # sas2ircu 0 display
> > > LSI Corporation SAS2 IR Configuration Utility.
> > > Version 18.00.00.00 (2013.11.18)
> > > Copyright (c) 2009-2013 LSI Corporation. All rights reserved.
> > >
> > > Read configuration has been initiated for controller 0
> > > --------------------------------------------------------------------
> > > --
> > > --
> > > Controller information
> > > ------------------------------------------------------------------------
> > > Controller type : SAS2008
> > > BIOS version : 7.35.00.00
> > > Firmware version : 18.00.00.00
> > > Channel description : 1 Serial Attached SCSI
> > > Initiator ID : 0
> > > Maximum physical devices : 255
> > > Concurrent commands supported : 3432
> > > Slot : 3
> > > Segment : 0
> > > Bus : 17
> > > Device : 0
> > > Function : 0
> > > RAID Support : No
> > > --------------------------------------------------------------------
> > > --
> > > --
> > > IR Volume information
> > > --------------------------------------------------------------------
> > > --
> > > --
> > > --------------------------------------------------------------------
> > > --
> > > --
> > > Physical device information
> > > --------------------------------------------------------------------
> > > --
> > > --
> > > Initiator at ID #0
> > >
> > > Device is a Hard disk
> > > Enclosure # : 2
> > > Slot # : 16
> > > SAS Address : 5000c50-0-05b5-ce25
> > > State : Ready (RDY)
> > > Size (in MB)/(in sectors) : 140014/286749479
> > > Manufacturer : SEAGATE
> > > Model Number : ST9146803SS
> > > Firmware Revision : FS03
> > > Serial No : 3SD02W5L
> > > GUID : N/A
> > > Protocol : SAS
> > > Drive Type : SAS_HDD
> > >
> > > Device is a Hard disk
> > > Enclosure # : 2
> > > Slot # : 17
> > > SAS Address : 5005076-0-3e8e-81a2
> > > State : Ready (RDY)
> > > Size (in MB)/(in sectors) : 953869/1953525167
> > > Manufacturer : ATA
> > > Model Number : Samsung SSD 840
> > > Firmware Revision : BB0Q
> > > Serial No : S1D9NEADA08549F
> > > GUID : N/A
> > > Protocol : SATA
> > > Drive Type : SATA_SSD
> > >
> > > Device is a Hard disk
> > > Enclosure # : 2
> > > Slot # : 18
> > > SAS Address : 5005076-0-3e8e-81a3
> > > State : Ready (RDY)
> > > Size (in MB)/(in sectors) : 953869/1953525167
> > > Manufacturer : ATA
> > > Model Number : Samsung SSD 840
> > > Firmware Revision : BB0Q
> > > Serial No : S1D9NEADA08548T
> > > GUID : N/A
> > > Protocol : SATA
> > > Drive Type : SATA_SSD
> > >
> > > Device is a Hard disk
> > > Enclosure # : 2
> > > Slot # : 19
> > > SAS Address : 5005076-0-3e8e-81a4
> > > State : Ready (RDY)
> > > Size (in MB)/(in sectors) : 953869/1953525167
> > > Manufacturer : ATA
> > > Model Number : Samsung SSD 840
> > > Firmware Revision : BB0Q
> > > Serial No : S1D9NEADA08568E
> > > GUID : N/A
> > > Protocol : SATA
> > > Drive Type : SATA_SSD
> > >
> > > Device is a Hard disk
> > > Enclosure # : 2
> > > Slot # : 20
> > > SAS Address : 5005076-0-3e8e-81a5
> > > State : Ready (RDY)
> > > Size (in MB)/(in sectors) : 953869/1953525167
> > > Manufacturer : ATA
> > > Model Number : Samsung SSD 840
> > > Firmware Revision : BB0Q
> > > Serial No : S1D9NEADA08547X
> > > GUID : N/A
> > > Protocol : SATA
> > > Drive Type : SATA_SSD
> > >
> > > Device is a Hard disk
> > > Enclosure # : 2
> > > Slot # : 21
> > > SAS Address : 5005076-0-3e8e-81a6
> > > State : Ready (RDY)
> > > Size (in MB)/(in sectors) : 953869/1953525167
> > > Manufacturer : ATA
> > > Model Number : Samsung SSD 840
> > > Firmware Revision : BB0Q
> > > Serial No : S1D9NEADA08518Y
> > > GUID : N/A
> > > Protocol : SATA
> > > Drive Type : SATA_SSD
> > >
> > > Device is a Hard disk
> > > Enclosure # : 2
> > > Slot # : 22
> > > SAS Address : 5005076-0-3e8e-81a7
> > > State : Ready (RDY)
> > > Size (in MB)/(in sectors) : 953869/1953525167
> > > Manufacturer : ATA
> > > Model Number : Samsung SSD 840
> > > Firmware Revision : BB0Q
> > > Serial No : S1D9NEADA08556K
> > > GUID : N/A
> > > Protocol : SATA
> > > Drive Type : SATA_SSD
> > >
> > > Device is a Enclosure services device
> > > Enclosure # : 2
> > > Slot # : 255
> > > SAS Address : 5005076-0-3e8e-81b9
> > > State : Standby (SBY)
> > > Manufacturer : IBM-ESXS
> > > Model Number : SAS EXP BP
> > > Firmware Revision : 61A6
> > > Serial No : 00000006
> > > GUID : N/A
> > > Protocol : SAS
> > > Device Type : Enclosure services device
> > >
> > > Device is a Hard disk
> > > Enclosure # : 3
> > > Slot # : 0
> > > SAS Address : 5005076-0-3e8e-86e9
> > > State : Ready (RDY)
> > > Size (in MB)/(in sectors) : 953869/1953525167
> > > Manufacturer : ATA
> > > Model Number : Samsung SSD 840
> > > Firmware Revision : BB0Q
> > > Serial No : S1D9NEADA08550R
> > > GUID : N/A
> > > Protocol : SATA
> > > Drive Type : SATA_SSD
> > >
> > > Device is a Hard disk
> > > Enclosure # : 3
> > > Slot # : 1
> > > SAS Address : 5005076-0-3e8e-86ea
> > > State : Ready (RDY)
> > > Size (in MB)/(in sectors) : 953869/1953525167
> > > Manufacturer : ATA
> > > Model Number : Samsung SSD 840
> > > Firmware Revision : BB0Q
> > > Serial No : S1D9NEADA08911Y
> > > GUID : N/A
> > > Protocol : SATA
> > > Drive Type : SATA_SSD
> > >
> > > Device is a Hard disk
> > > Enclosure # : 3
> > > Slot # : 2
> > > SAS Address : 5005076-0-3e8e-86eb
> > > State : Ready (RDY)
> > > Size (in MB)/(in sectors) : 953869/1953525167
> > > Manufacturer : ATA
> > > Model Number : Samsung SSD 840
> > > Firmware Revision : BB0Q
> > > Serial No : S1D9NEADA08811L
> > > GUID : N/A
> > > Protocol : SATA
> > > Drive Type : SATA_SSD
> > >
> > > Device is a Hard disk
> > > Enclosure # : 3
> > > Slot # : 13
> > > SAS Address : 5000c50-0-05b5-e531
> > > State : Ready (RDY)
> > > Size (in MB)/(in sectors) : 140014/286749479
> > > Manufacturer : SEAGATE
> > > Model Number : ST9146803SS
> > > Firmware Revision : FS03
> > > Serial No : 3SD02STR
> > > GUID : N/A
> > > Protocol : SAS
> > > Drive Type : SAS_HDD
> > >
> > > Device is a Hard disk
> > > Enclosure # : 3
> > > Slot # : 14
> > > SAS Address : 5000c50-0-05b5-d489
> > > State : Ready (RDY)
> > > Size (in MB)/(in sectors) : 140014/286749479
> > > Manufacturer : SEAGATE
> > > Model Number : ST9146803SS
> > > Firmware Revision : FS03
> > > Serial No : 3SD02TV1
> > > GUID : N/A
> > > Protocol : SAS
> > > Drive Type : SAS_HDD
> > >
> > > Device is a Hard disk
> > > Enclosure # : 3
> > > Slot # : 15
> > > SAS Address : 5000c50-0-05b5-f0ad
> > > State : Ready (RDY)
> > > Size (in MB)/(in sectors) : 140014/286749479
> > > Manufacturer : SEAGATE
> > > Model Number : ST9146803SS
> > > Firmware Revision : FS03
> > > Serial No : 3SD03F4C
> > > GUID : N/A
> > > Protocol : SAS
> > > Drive Type : SAS_HDD
> > >
> > > Device is a Enclosure services device
> > > Enclosure # : 3
> > > Slot # : 255
> > > SAS Address : 5005076-0-3e8e-86f9
> > > State : Standby (SBY)
> > > Manufacturer : IBM-ESXS
> > > Model Number : SAS EXP BP
> > > Firmware Revision : 61A6
> > > Serial No : 00000006
> > > GUID : N/A
> > > Protocol : SAS
> > > Device Type : Enclosure services device
> > > --------------------------------------------------------------------
> > > --
> > > --
> > > Enclosure information
> > > ------------------------------------------------------------------------
> > > Enclosure# : 1
> > > Logical ID : 500605b0:07ba2100
> > > Numslots : 8
> > > StartSlot : 0
> > > Enclosure# : 2
> > > Logical ID : 50050760:3e8e81a0
> > > Numslots : 25
> > > StartSlot : 0
> > > Enclosure# : 3
> > > Logical ID : 50050760:3e8e86e0
> > > Numslots : 25
> > > StartSlot : 0
> > > --------------------------------------------------------------------
> > > --
> > > --
> > > SAS2IRCU: Command DISPLAY Completed Successfully.
> > > SAS2IRCU: Utility Completed Successfully.
> > >
> > >
> > >
> > >
> > > _______________________________________________
> > > freebsd-scsi at freebsd.org mailing list
> > > http://lists.freebsd.org/mailman/listinfo/freebsd-scsi
> > > To unsubscribe, send any mail to
> > > "freebsd-scsi-unsubscribe at freebsd.org"
> >
> > Adding Steve McConnell, who may not be on the freebsd-scsi list.
>
> Right, I'm not on the freebsd-scsi list. I suspect that the reason for
> this is that
> most of the Gen3 performance gains come from MSI-x support, but that is
> not in
> the mpr driver yet, and MSI-x is not supported in the 2008 card. But, I
> have a
> question out to another person and I'm waiting for more info.
>
> Steve
Borja, can you upgrade your card to the latest FW. Your version is really
old (1.00.03.00). We should be up to 7.00.00.00 now. There could be a lot
of fixes that would affect performance.
Steve
More information about the freebsd-scsi
mailing list