mpr vs mps performance
Borja Marcos
borjam at sarenet.es
Mon Sep 22 15:47:01 UTC 2014
Hello,
I have been playing with the new SAS3 cards supported by the mpr driver, and I´ve found out that they are, in the same hardware
configuration, considerably slower writing data. Moreover, running two simultaneous "bonnie" benchmarks (I am using SSDs, and one "bonnie" sometimes hits 100% CPU usage, unable to really saturate the I/O) I see the writing activity somewhat stalling, with disk bandwidth going from 600 MB/s to around 50 for 20 seconds or so.
I'd like to know if this matches anyone else's experiences. Also, I can try and make some tests if needed. But for now it seems we will stick to the SAS2 HBAs.
The Bonnie results are:
With mpr driver, SAS3:
(each bonnie instance, so multiply the results by 2 to get the actual bandwidth achieved)
Seq output: (writing)
Block: 292155 KB/s
Rewrite: 139713 KB/s
Seq input:
Block: 862861 KB/s
With mps driver: SAS2, again, total is 2x the following figures.
Seq output: (writing)
Block: 587950 KB/s
Rewrite: 208239 KB/s
Seq. input: (reading)
Block: 842169 KB/s
The storage is a ZFS pool with a 9-disk raidz2 vdev, made of Samsung 840 EVO 1 TB SSDs. The pool has been created with an ashift of 12 (zpool applied it thanks to the 4 KB block quirk for these SSDs)
<ATA Samsung SSD 840 BB0Q> at scbus0 target 9 lun 0 (pass0,da0)
<ATA Samsung SSD 840 BB0Q> at scbus0 target 10 lun 0 (pass1,da1)
<ATA Samsung SSD 840 BB0Q> at scbus0 target 11 lun 0 (pass2,da2)
<ATA Samsung SSD 840 BB0Q> at scbus0 target 12 lun 0 (pass3,da3)
<ATA Samsung SSD 840 BB0Q> at scbus0 target 13 lun 0 (pass4,da4)
<ATA Samsung SSD 840 BB0Q> at scbus0 target 14 lun 0 (pass5,da5)
<ATA Samsung SSD 840 BB0Q> at scbus0 target 17 lun 0 (pass7,da6)
<ATA Samsung SSD 840 BB0Q> at scbus0 target 18 lun 0 (pass8,da7)
<ATA Samsung SSD 840 BB0Q> at scbus0 target 27 lun 0 (pass14,da12)
The mpr card details follow:
Sep 17 09:49:39 elibm kernel: mpr0: <LSI SAS3008> port 0x3f00-0x3fff mem 0x912f0000-0x912fffff irq 32 at device 0.0 on pci17
Sep 17 09:49:39 elibm kernel: mpr0: IOCFacts :
Sep 17 09:49:39 elibm kernel: MsgVersion: 0x205
Sep 17 09:49:39 elibm kernel: HeaderVersion: 0x1d00
Sep 17 09:49:39 elibm kernel: IOCNumber: 0
Sep 17 09:49:39 elibm kernel: IOCExceptions: 0x0
Sep 17 09:49:39 elibm kernel: MaxChainDepth: 128
Sep 17 09:49:39 elibm kernel: NumberOfPorts: 1
Sep 17 09:49:39 elibm kernel: RequestCredit: 11264
Sep 17 09:49:39 elibm kernel: ProductID: 0x2221
Sep 17 09:49:39 elibm kernel: IOCRequestFrameSize: 32
Sep 17 09:49:39 elibm kernel: MaxInitiators: 1
Sep 17 09:49:39 elibm kernel: MaxTargets: 1024
Sep 17 09:49:39 elibm kernel: MaxSasExpanders: 14
Sep 17 09:49:39 elibm kernel: MaxEnclosures: 15
Sep 17 09:49:39 elibm kernel: HighPriorityCredit: 60
Sep 17 09:49:39 elibm kernel: MaxReplyDescriptorPostQueueDepth: 65504
Sep 17 09:49:39 elibm kernel: ReplyFrameSize: 32
Sep 17 09:49:39 elibm kernel: MaxVolumes: 0
Sep 17 09:49:39 elibm kernel: MaxDevHandle: 1047
Sep 17 09:49:39 elibm kernel: MaxPersistentEntries: 128
Sep 17 09:49:39 elibm kernel: mpr0: Firmware: 01.00.03.00, Driver: 05.255.05.00-fbsd
Sep 17 09:49:39 elibm kernel: mpr0: IOCCapabilities: 3a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc>
And the mps card is a classic:
Sep 22 17:18:24 elibm kernel: mps0: <LSI SAS2008> port 0x3f00-0x3fff mem 0x90ebc000-0x90ebffff,0x912c0000-0x912fffff irq 32 at device 0.0 on pci17
Sep 22 17:18:24 elibm kernel: mps0: Firmware: 18.00.00.00, Driver: 19.00.00.00-fbsd
Sep 22 17:18:24 elibm kernel: mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
The connected devices follow. Both use the same hardware (except for the cables and HBA of course), but currently there's no way to check this with the SAS3 card, as sas3ircu nor sas3flash detect it on FreeBSD.
# sas2ircu 0 display
LSI Corporation SAS2 IR Configuration Utility.
Version 18.00.00.00 (2013.11.18)
Copyright (c) 2009-2013 LSI Corporation. All rights reserved.
Read configuration has been initiated for controller 0
------------------------------------------------------------------------
Controller information
------------------------------------------------------------------------
Controller type : SAS2008
BIOS version : 7.35.00.00
Firmware version : 18.00.00.00
Channel description : 1 Serial Attached SCSI
Initiator ID : 0
Maximum physical devices : 255
Concurrent commands supported : 3432
Slot : 3
Segment : 0
Bus : 17
Device : 0
Function : 0
RAID Support : No
------------------------------------------------------------------------
IR Volume information
------------------------------------------------------------------------
------------------------------------------------------------------------
Physical device information
------------------------------------------------------------------------
Initiator at ID #0
Device is a Hard disk
Enclosure # : 2
Slot # : 16
SAS Address : 5000c50-0-05b5-ce25
State : Ready (RDY)
Size (in MB)/(in sectors) : 140014/286749479
Manufacturer : SEAGATE
Model Number : ST9146803SS
Firmware Revision : FS03
Serial No : 3SD02W5L
GUID : N/A
Protocol : SAS
Drive Type : SAS_HDD
Device is a Hard disk
Enclosure # : 2
Slot # : 17
SAS Address : 5005076-0-3e8e-81a2
State : Ready (RDY)
Size (in MB)/(in sectors) : 953869/1953525167
Manufacturer : ATA
Model Number : Samsung SSD 840
Firmware Revision : BB0Q
Serial No : S1D9NEADA08549F
GUID : N/A
Protocol : SATA
Drive Type : SATA_SSD
Device is a Hard disk
Enclosure # : 2
Slot # : 18
SAS Address : 5005076-0-3e8e-81a3
State : Ready (RDY)
Size (in MB)/(in sectors) : 953869/1953525167
Manufacturer : ATA
Model Number : Samsung SSD 840
Firmware Revision : BB0Q
Serial No : S1D9NEADA08548T
GUID : N/A
Protocol : SATA
Drive Type : SATA_SSD
Device is a Hard disk
Enclosure # : 2
Slot # : 19
SAS Address : 5005076-0-3e8e-81a4
State : Ready (RDY)
Size (in MB)/(in sectors) : 953869/1953525167
Manufacturer : ATA
Model Number : Samsung SSD 840
Firmware Revision : BB0Q
Serial No : S1D9NEADA08568E
GUID : N/A
Protocol : SATA
Drive Type : SATA_SSD
Device is a Hard disk
Enclosure # : 2
Slot # : 20
SAS Address : 5005076-0-3e8e-81a5
State : Ready (RDY)
Size (in MB)/(in sectors) : 953869/1953525167
Manufacturer : ATA
Model Number : Samsung SSD 840
Firmware Revision : BB0Q
Serial No : S1D9NEADA08547X
GUID : N/A
Protocol : SATA
Drive Type : SATA_SSD
Device is a Hard disk
Enclosure # : 2
Slot # : 21
SAS Address : 5005076-0-3e8e-81a6
State : Ready (RDY)
Size (in MB)/(in sectors) : 953869/1953525167
Manufacturer : ATA
Model Number : Samsung SSD 840
Firmware Revision : BB0Q
Serial No : S1D9NEADA08518Y
GUID : N/A
Protocol : SATA
Drive Type : SATA_SSD
Device is a Hard disk
Enclosure # : 2
Slot # : 22
SAS Address : 5005076-0-3e8e-81a7
State : Ready (RDY)
Size (in MB)/(in sectors) : 953869/1953525167
Manufacturer : ATA
Model Number : Samsung SSD 840
Firmware Revision : BB0Q
Serial No : S1D9NEADA08556K
GUID : N/A
Protocol : SATA
Drive Type : SATA_SSD
Device is a Enclosure services device
Enclosure # : 2
Slot # : 255
SAS Address : 5005076-0-3e8e-81b9
State : Standby (SBY)
Manufacturer : IBM-ESXS
Model Number : SAS EXP BP
Firmware Revision : 61A6
Serial No : 00000006
GUID : N/A
Protocol : SAS
Device Type : Enclosure services device
Device is a Hard disk
Enclosure # : 3
Slot # : 0
SAS Address : 5005076-0-3e8e-86e9
State : Ready (RDY)
Size (in MB)/(in sectors) : 953869/1953525167
Manufacturer : ATA
Model Number : Samsung SSD 840
Firmware Revision : BB0Q
Serial No : S1D9NEADA08550R
GUID : N/A
Protocol : SATA
Drive Type : SATA_SSD
Device is a Hard disk
Enclosure # : 3
Slot # : 1
SAS Address : 5005076-0-3e8e-86ea
State : Ready (RDY)
Size (in MB)/(in sectors) : 953869/1953525167
Manufacturer : ATA
Model Number : Samsung SSD 840
Firmware Revision : BB0Q
Serial No : S1D9NEADA08911Y
GUID : N/A
Protocol : SATA
Drive Type : SATA_SSD
Device is a Hard disk
Enclosure # : 3
Slot # : 2
SAS Address : 5005076-0-3e8e-86eb
State : Ready (RDY)
Size (in MB)/(in sectors) : 953869/1953525167
Manufacturer : ATA
Model Number : Samsung SSD 840
Firmware Revision : BB0Q
Serial No : S1D9NEADA08811L
GUID : N/A
Protocol : SATA
Drive Type : SATA_SSD
Device is a Hard disk
Enclosure # : 3
Slot # : 13
SAS Address : 5000c50-0-05b5-e531
State : Ready (RDY)
Size (in MB)/(in sectors) : 140014/286749479
Manufacturer : SEAGATE
Model Number : ST9146803SS
Firmware Revision : FS03
Serial No : 3SD02STR
GUID : N/A
Protocol : SAS
Drive Type : SAS_HDD
Device is a Hard disk
Enclosure # : 3
Slot # : 14
SAS Address : 5000c50-0-05b5-d489
State : Ready (RDY)
Size (in MB)/(in sectors) : 140014/286749479
Manufacturer : SEAGATE
Model Number : ST9146803SS
Firmware Revision : FS03
Serial No : 3SD02TV1
GUID : N/A
Protocol : SAS
Drive Type : SAS_HDD
Device is a Hard disk
Enclosure # : 3
Slot # : 15
SAS Address : 5000c50-0-05b5-f0ad
State : Ready (RDY)
Size (in MB)/(in sectors) : 140014/286749479
Manufacturer : SEAGATE
Model Number : ST9146803SS
Firmware Revision : FS03
Serial No : 3SD03F4C
GUID : N/A
Protocol : SAS
Drive Type : SAS_HDD
Device is a Enclosure services device
Enclosure # : 3
Slot # : 255
SAS Address : 5005076-0-3e8e-86f9
State : Standby (SBY)
Manufacturer : IBM-ESXS
Model Number : SAS EXP BP
Firmware Revision : 61A6
Serial No : 00000006
GUID : N/A
Protocol : SAS
Device Type : Enclosure services device
------------------------------------------------------------------------
Enclosure information
------------------------------------------------------------------------
Enclosure# : 1
Logical ID : 500605b0:07ba2100
Numslots : 8
StartSlot : 0
Enclosure# : 2
Logical ID : 50050760:3e8e81a0
Numslots : 25
StartSlot : 0
Enclosure# : 3
Logical ID : 50050760:3e8e86e0
Numslots : 25
StartSlot : 0
------------------------------------------------------------------------
SAS2IRCU: Command DISPLAY Completed Successfully.
SAS2IRCU: Utility Completed Successfully.
More information about the freebsd-scsi
mailing list