mpr vs mps performance
Alan Somers
asomers at freebsd.org
Mon Oct 6 16:13:09 UTC 2014
On Mon, Sep 22, 2014 at 9:39 AM, Borja Marcos <borjam at sarenet.es> wrote:
>
> Hello,
>
> I have been playing with the new SAS3 cards supported by the mpr driver, and I´ve found out that they are, in the same hardware
> configuration, considerably slower writing data. Moreover, running two simultaneous "bonnie" benchmarks (I am using SSDs, and one "bonnie" sometimes hits 100% CPU usage, unable to really saturate the I/O) I see the writing activity somewhat stalling, with disk bandwidth going from 600 MB/s to around 50 for 20 seconds or so.
>
> I'd like to know if this matches anyone else's experiences. Also, I can try and make some tests if needed. But for now it seems we will stick to the SAS2 HBAs.
>
>
>
>
> The Bonnie results are:
>
> With mpr driver, SAS3:
>
> (each bonnie instance, so multiply the results by 2 to get the actual bandwidth achieved)
>
> Seq output: (writing)
> Block: 292155 KB/s
> Rewrite: 139713 KB/s
> Seq input:
> Block: 862861 KB/s
>
> With mps driver: SAS2, again, total is 2x the following figures.
>
> Seq output: (writing)
> Block: 587950 KB/s
> Rewrite: 208239 KB/s
> Seq. input: (reading)
> Block: 842169 KB/s
>
>
> The storage is a ZFS pool with a 9-disk raidz2 vdev, made of Samsung 840 EVO 1 TB SSDs. The pool has been created with an ashift of 12 (zpool applied it thanks to the 4 KB block quirk for these SSDs)
>
> <ATA Samsung SSD 840 BB0Q> at scbus0 target 9 lun 0 (pass0,da0)
> <ATA Samsung SSD 840 BB0Q> at scbus0 target 10 lun 0 (pass1,da1)
> <ATA Samsung SSD 840 BB0Q> at scbus0 target 11 lun 0 (pass2,da2)
> <ATA Samsung SSD 840 BB0Q> at scbus0 target 12 lun 0 (pass3,da3)
> <ATA Samsung SSD 840 BB0Q> at scbus0 target 13 lun 0 (pass4,da4)
> <ATA Samsung SSD 840 BB0Q> at scbus0 target 14 lun 0 (pass5,da5)
> <ATA Samsung SSD 840 BB0Q> at scbus0 target 17 lun 0 (pass7,da6)
> <ATA Samsung SSD 840 BB0Q> at scbus0 target 18 lun 0 (pass8,da7)
> <ATA Samsung SSD 840 BB0Q> at scbus0 target 27 lun 0 (pass14,da12)
>
>
>
>
> The mpr card details follow:
> Sep 17 09:49:39 elibm kernel: mpr0: <LSI SAS3008> port 0x3f00-0x3fff mem 0x912f0000-0x912fffff irq 32 at device 0.0 on pci17
> Sep 17 09:49:39 elibm kernel: mpr0: IOCFacts :
> Sep 17 09:49:39 elibm kernel: MsgVersion: 0x205
> Sep 17 09:49:39 elibm kernel: HeaderVersion: 0x1d00
> Sep 17 09:49:39 elibm kernel: IOCNumber: 0
> Sep 17 09:49:39 elibm kernel: IOCExceptions: 0x0
> Sep 17 09:49:39 elibm kernel: MaxChainDepth: 128
> Sep 17 09:49:39 elibm kernel: NumberOfPorts: 1
> Sep 17 09:49:39 elibm kernel: RequestCredit: 11264
> Sep 17 09:49:39 elibm kernel: ProductID: 0x2221
> Sep 17 09:49:39 elibm kernel: IOCRequestFrameSize: 32
> Sep 17 09:49:39 elibm kernel: MaxInitiators: 1
> Sep 17 09:49:39 elibm kernel: MaxTargets: 1024
> Sep 17 09:49:39 elibm kernel: MaxSasExpanders: 14
> Sep 17 09:49:39 elibm kernel: MaxEnclosures: 15
> Sep 17 09:49:39 elibm kernel: HighPriorityCredit: 60
> Sep 17 09:49:39 elibm kernel: MaxReplyDescriptorPostQueueDepth: 65504
> Sep 17 09:49:39 elibm kernel: ReplyFrameSize: 32
> Sep 17 09:49:39 elibm kernel: MaxVolumes: 0
> Sep 17 09:49:39 elibm kernel: MaxDevHandle: 1047
> Sep 17 09:49:39 elibm kernel: MaxPersistentEntries: 128
> Sep 17 09:49:39 elibm kernel: mpr0: Firmware: 01.00.03.00, Driver: 05.255.05.00-fbsd
> Sep 17 09:49:39 elibm kernel: mpr0: IOCCapabilities: 3a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc>
>
>
> And the mps card is a classic:
> Sep 22 17:18:24 elibm kernel: mps0: <LSI SAS2008> port 0x3f00-0x3fff mem 0x90ebc000-0x90ebffff,0x912c0000-0x912fffff irq 32 at device 0.0 on pci17
> Sep 22 17:18:24 elibm kernel: mps0: Firmware: 18.00.00.00, Driver: 19.00.00.00-fbsd
> Sep 22 17:18:24 elibm kernel: mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
>
>
> The connected devices follow. Both use the same hardware (except for the cables and HBA of course), but currently there's no way to check this with the SAS3 card, as sas3ircu nor sas3flash detect it on FreeBSD.
>
>
>
> # sas2ircu 0 display
> LSI Corporation SAS2 IR Configuration Utility.
> Version 18.00.00.00 (2013.11.18)
> Copyright (c) 2009-2013 LSI Corporation. All rights reserved.
>
> Read configuration has been initiated for controller 0
> ------------------------------------------------------------------------
> Controller information
> ------------------------------------------------------------------------
> Controller type : SAS2008
> BIOS version : 7.35.00.00
> Firmware version : 18.00.00.00
> Channel description : 1 Serial Attached SCSI
> Initiator ID : 0
> Maximum physical devices : 255
> Concurrent commands supported : 3432
> Slot : 3
> Segment : 0
> Bus : 17
> Device : 0
> Function : 0
> RAID Support : No
> ------------------------------------------------------------------------
> IR Volume information
> ------------------------------------------------------------------------
> ------------------------------------------------------------------------
> Physical device information
> ------------------------------------------------------------------------
> Initiator at ID #0
>
> Device is a Hard disk
> Enclosure # : 2
> Slot # : 16
> SAS Address : 5000c50-0-05b5-ce25
> State : Ready (RDY)
> Size (in MB)/(in sectors) : 140014/286749479
> Manufacturer : SEAGATE
> Model Number : ST9146803SS
> Firmware Revision : FS03
> Serial No : 3SD02W5L
> GUID : N/A
> Protocol : SAS
> Drive Type : SAS_HDD
>
> Device is a Hard disk
> Enclosure # : 2
> Slot # : 17
> SAS Address : 5005076-0-3e8e-81a2
> State : Ready (RDY)
> Size (in MB)/(in sectors) : 953869/1953525167
> Manufacturer : ATA
> Model Number : Samsung SSD 840
> Firmware Revision : BB0Q
> Serial No : S1D9NEADA08549F
> GUID : N/A
> Protocol : SATA
> Drive Type : SATA_SSD
>
> Device is a Hard disk
> Enclosure # : 2
> Slot # : 18
> SAS Address : 5005076-0-3e8e-81a3
> State : Ready (RDY)
> Size (in MB)/(in sectors) : 953869/1953525167
> Manufacturer : ATA
> Model Number : Samsung SSD 840
> Firmware Revision : BB0Q
> Serial No : S1D9NEADA08548T
> GUID : N/A
> Protocol : SATA
> Drive Type : SATA_SSD
>
> Device is a Hard disk
> Enclosure # : 2
> Slot # : 19
> SAS Address : 5005076-0-3e8e-81a4
> State : Ready (RDY)
> Size (in MB)/(in sectors) : 953869/1953525167
> Manufacturer : ATA
> Model Number : Samsung SSD 840
> Firmware Revision : BB0Q
> Serial No : S1D9NEADA08568E
> GUID : N/A
> Protocol : SATA
> Drive Type : SATA_SSD
>
> Device is a Hard disk
> Enclosure # : 2
> Slot # : 20
> SAS Address : 5005076-0-3e8e-81a5
> State : Ready (RDY)
> Size (in MB)/(in sectors) : 953869/1953525167
> Manufacturer : ATA
> Model Number : Samsung SSD 840
> Firmware Revision : BB0Q
> Serial No : S1D9NEADA08547X
> GUID : N/A
> Protocol : SATA
> Drive Type : SATA_SSD
>
> Device is a Hard disk
> Enclosure # : 2
> Slot # : 21
> SAS Address : 5005076-0-3e8e-81a6
> State : Ready (RDY)
> Size (in MB)/(in sectors) : 953869/1953525167
> Manufacturer : ATA
> Model Number : Samsung SSD 840
> Firmware Revision : BB0Q
> Serial No : S1D9NEADA08518Y
> GUID : N/A
> Protocol : SATA
> Drive Type : SATA_SSD
>
> Device is a Hard disk
> Enclosure # : 2
> Slot # : 22
> SAS Address : 5005076-0-3e8e-81a7
> State : Ready (RDY)
> Size (in MB)/(in sectors) : 953869/1953525167
> Manufacturer : ATA
> Model Number : Samsung SSD 840
> Firmware Revision : BB0Q
> Serial No : S1D9NEADA08556K
> GUID : N/A
> Protocol : SATA
> Drive Type : SATA_SSD
>
> Device is a Enclosure services device
> Enclosure # : 2
> Slot # : 255
> SAS Address : 5005076-0-3e8e-81b9
> State : Standby (SBY)
> Manufacturer : IBM-ESXS
> Model Number : SAS EXP BP
> Firmware Revision : 61A6
> Serial No : 00000006
> GUID : N/A
> Protocol : SAS
> Device Type : Enclosure services device
>
> Device is a Hard disk
> Enclosure # : 3
> Slot # : 0
> SAS Address : 5005076-0-3e8e-86e9
> State : Ready (RDY)
> Size (in MB)/(in sectors) : 953869/1953525167
> Manufacturer : ATA
> Model Number : Samsung SSD 840
> Firmware Revision : BB0Q
> Serial No : S1D9NEADA08550R
> GUID : N/A
> Protocol : SATA
> Drive Type : SATA_SSD
>
> Device is a Hard disk
> Enclosure # : 3
> Slot # : 1
> SAS Address : 5005076-0-3e8e-86ea
> State : Ready (RDY)
> Size (in MB)/(in sectors) : 953869/1953525167
> Manufacturer : ATA
> Model Number : Samsung SSD 840
> Firmware Revision : BB0Q
> Serial No : S1D9NEADA08911Y
> GUID : N/A
> Protocol : SATA
> Drive Type : SATA_SSD
>
> Device is a Hard disk
> Enclosure # : 3
> Slot # : 2
> SAS Address : 5005076-0-3e8e-86eb
> State : Ready (RDY)
> Size (in MB)/(in sectors) : 953869/1953525167
> Manufacturer : ATA
> Model Number : Samsung SSD 840
> Firmware Revision : BB0Q
> Serial No : S1D9NEADA08811L
> GUID : N/A
> Protocol : SATA
> Drive Type : SATA_SSD
>
> Device is a Hard disk
> Enclosure # : 3
> Slot # : 13
> SAS Address : 5000c50-0-05b5-e531
> State : Ready (RDY)
> Size (in MB)/(in sectors) : 140014/286749479
> Manufacturer : SEAGATE
> Model Number : ST9146803SS
> Firmware Revision : FS03
> Serial No : 3SD02STR
> GUID : N/A
> Protocol : SAS
> Drive Type : SAS_HDD
>
> Device is a Hard disk
> Enclosure # : 3
> Slot # : 14
> SAS Address : 5000c50-0-05b5-d489
> State : Ready (RDY)
> Size (in MB)/(in sectors) : 140014/286749479
> Manufacturer : SEAGATE
> Model Number : ST9146803SS
> Firmware Revision : FS03
> Serial No : 3SD02TV1
> GUID : N/A
> Protocol : SAS
> Drive Type : SAS_HDD
>
> Device is a Hard disk
> Enclosure # : 3
> Slot # : 15
> SAS Address : 5000c50-0-05b5-f0ad
> State : Ready (RDY)
> Size (in MB)/(in sectors) : 140014/286749479
> Manufacturer : SEAGATE
> Model Number : ST9146803SS
> Firmware Revision : FS03
> Serial No : 3SD03F4C
> GUID : N/A
> Protocol : SAS
> Drive Type : SAS_HDD
>
> Device is a Enclosure services device
> Enclosure # : 3
> Slot # : 255
> SAS Address : 5005076-0-3e8e-86f9
> State : Standby (SBY)
> Manufacturer : IBM-ESXS
> Model Number : SAS EXP BP
> Firmware Revision : 61A6
> Serial No : 00000006
> GUID : N/A
> Protocol : SAS
> Device Type : Enclosure services device
> ------------------------------------------------------------------------
> Enclosure information
> ------------------------------------------------------------------------
> Enclosure# : 1
> Logical ID : 500605b0:07ba2100
> Numslots : 8
> StartSlot : 0
> Enclosure# : 2
> Logical ID : 50050760:3e8e81a0
> Numslots : 25
> StartSlot : 0
> Enclosure# : 3
> Logical ID : 50050760:3e8e86e0
> Numslots : 25
> StartSlot : 0
> ------------------------------------------------------------------------
> SAS2IRCU: Command DISPLAY Completed Successfully.
> SAS2IRCU: Utility Completed Successfully.
>
>
>
>
> _______________________________________________
> freebsd-scsi at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-scsi
> To unsubscribe, send any mail to "freebsd-scsi-unsubscribe at freebsd.org"
Adding Steve McConnell, who may not be on the freebsd-scsi list.
More information about the freebsd-scsi
mailing list