Poor SSD performance with Dell PERC H730 Mini attached to the mfi(4) driver on FreeBSD 10.2
Pallav Bose
pallav_bose at yahoo.com
Tue Oct 13 02:45:03 UTC 2015
+freebsd-scsi, Sibananda
MFI_FUSION_ENABLE_INTERRUPT_MASK has been defined differently in mfi(4) vs. mrsas(4) - 0x00000008 vs. 0x00000009. The mrsas(4) #define matches the Linux version. Is there a bug in the way the ISR is being setup in case of mfi(4) + Invader card?
When I print the value of "status" from the function mfi_tbolt_check_clear_intr_ppc(), it equals 0x40000008. Please note that during this time there is no disk activity, but interrupts are continuously being generated (when I check vmstat -i).int32_t
mfi_tbolt_check_clear_intr_ppc(struct mfi_softc *sc)
{
int32_t status, mfi_status = 0;
status = MFI_READ4(sc, MFI_OSTS);
if (status & 1) {
MFI_WRITE4(sc, MFI_OSTS, status);
MFI_READ4(sc, MFI_OSTS);
if (status & MFI_STATE_CHANGE_INTERRUPT) {
mfi_status |= MFI_FIRMWARE_STATE_CHANGE;
}
return mfi_status;
}
if (!(status & MFI_FUSION_ENABLE_INTERRUPT_MASK))
return 1;
MFI_READ4(sc, MFI_OSTS);
return 0;
}So, mfi_tbolt_check_clear_intr_ppc() returns 0 to the caller mfi_intr_tbolt() even though the interrupt was a spurious one?
Regards,
Pallav
On Monday, October 5, 2015 3:32 PM, Pallav Bose <pallav_bose at yahoo.com> wrote:
Hi,
Hello,
I have two Dell PowerEdge R430 servers - dell13g-mfi and dell13g-mrsas - both with PERC H730 Mini RAID controllers. I have FreeBSD 10.2 running on each server. One server is using the mfi(4) driver, and the other the mrsas(4) driver. I'm observing very poor SSD throughput on the server using the mfi(4) driver. I saw similarly poor throughput on FreeBSD 8.0 with a patched mfi(4) driver and posted a question (https://lists.freebsd.org/pipermail/freebsd-questions/2015-September/268349.html), so I tried FreeBSD 10.2 and see similar results.
dell13g-mfi info:
root at dell13g-mfi:~ # uname -a
FreeBSD dell13g-mfi 10.2-RELEASE FreeBSD 10.2-RELEASE #0 r286666: Wed Aug 12 15:26:37 UTC 2015 root at releng1.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
root at dell13g-mfi:~ # cat /boot/loader.conf
kern.geom.label.gptid.enable="0"
zfs_load="YES"
root at dell13g-mfi:~ # pciconf -bclveV mfi0
mfi0 at pci0:1:0:0: class=0x010400 card=0x1f491028 chip=0x005d1000 rev=0x02 hdr=0x00
vendor = 'LSI Logic / Symbios Logic'
device = 'MegaRAID SAS-3 3108 [Invader]'
class = mass storage
subclass = RAID
bar [10] = type I/O Port, range 32, base 0x2000, size 256, enabled
bar [14] = type Memory, range 64, base 0x91d00000, size 65536, enabled
bar [1c] = type Memory, range 64, base 0x91c00000, size 1048576, enabled
cap 01[50] = powerspec 3 supports D0 D1 D2 D3 current D0
cap 10[68] = PCI-Express 2 endpoint max data 256(4096) FLR link x8(x8)
speed 8.0(8.0) ASPM disabled(L0s)
cap 03[d0] = VPD
cap 05[a8] = MSI supports 1 message, 64 bit, vector masks enabled with 1 message
cap 11[c0] = MSI-X supports 97 messages
Table in map 0x14[0xe000], PBA in map 0x14[0xf000]
ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected
ecap 0019[1e0] = PCIe Sec 1 lane errors 0
ecap 0004[1c0] = Power Budgeting 1
ecap 000e[148] = ARI 1
PCI-e errors = Correctable Error Detected
Unsupported Request Detected
Corrected = Advisory Non-Fatal Error
dell13g-mrsas info:
root at dell13g-mrsas:~ # uname -a
FreeBSD dell13g-mrsas 10.2-RELEASE FreeBSD 10.2-RELEASE #0 r286666: Wed Aug 12 15:26:37 UTC 2015 root at releng1.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
root at dell13g-mrsas:~ # cat /boot/loader.conf
kern.geom.label.gptid.enable="0"
zfs_load="YES"
hw.mfi.mrsas_enable=1
root at dell13g-mrsas:~ # pciconf -bclveV mrsas0
mrsas0 at pci0:1:0:0: class=0x010400 card=0x1f491028 chip=0x005d1000 rev=0x02 hdr=0x00
vendor = 'LSI Logic / Symbios Logic'
device = 'MegaRAID SAS-3 3108 [Invader]'
class = mass storage
subclass = RAID
bar [10] = type I/O Port, range 32, base 0x2000, size 256, enabled
bar [14] = type Memory, range 64, base 0x91d00000, size 65536, enabled
bar [1c] = type Memory, range 64, base 0x91c00000, size 1048576, enabled
cap 01[50] = powerspec 3 supports D0 D1 D2 D3 current D0
cap 10[68] = PCI-Express 2 endpoint max data 256(4096) FLR link x8(x8)
speed 8.0(8.0) ASPM disabled(L0s)
cap 03[d0] = VPD
cap 05[a8] = MSI supports 1 message, 64 bit, vector masks
cap 11[c0] = MSI-X supports 97 messages, enabled
Table in map 0x14[0xe000], PBA in map 0x14[0xf000]
ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected
ecap 0019[1e0] = PCIe Sec 1 lane errors 0
ecap 0004[1c0] = Power Budgeting 1
ecap 000e[148] = ARI 1
PCI-e errors = Correctable Error Detected
Unsupported Request Detected
Corrected = Advisory Non-Fatal Error
A simple read from one of the SSDs shows very poor performance in case of mfi(4).
root at dell13g-mfi:~ # dd if=/dev/mfid1 of=/dev/null bs=1m count=100
100+0 records in
100+0 records out
104857600 bytes transferred in 19.956666 secs (5254264 bytes/sec)
Compare that with mrsas(4).
root at dell13g-mrsas:~ # dd if=/dev/da1 of=/dev/null bs=1m count=100
100+0 records in
100+0 records out
104857600 bytes transferred in 0.248799 secs (421455329 bytes/sec)
High CPU usage by swi6 on the server using mfi(4).
root at dell13g-mfi:~ # top -PSHI
last pid: 1779; load averages: 0.55, 0.50, 0.44 up 0+06:39:27 14:49:45
244 processes: 7 running, 223 sleeping, 14 waiting
CPU 0: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
CPU 1: 0.0% user, 0.0% nice, 0.0% system, 64.6% interrupt, 35.4% idle
CPU 2: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
CPU 3: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
CPU 4: 0.0% user, 0.0% nice, 0.4% system, 0.0% interrupt, 99.6% idle
CPU 5: 0.0% user, 0.0% nice, 0.0% system, 5.8% interrupt, 94.2% idle
Mem: 3256K Active, 25M Inact, 588M Wired, 30G Free
ARC: 207M Total, 51M MFU, 147M MRU, 16K Anon, 1864K Header, 7317K Other
Swap: 8192M Total, 8192M Free
PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
11 root 155 ki31 0K 96K CPU0 0 395:01 100.00% idle{idle: cpu0}
11 root 155 ki31 0K 96K CPU3 3 361:02 100.00% idle{idle: cpu3}
11 root 155 ki31 0K 96K CPU2 2 357:53 100.00% idle{idle: cpu2}
11 root 155 ki31 0K 96K RUN 4 358:00 99.85% idle{idle: cpu4}
11 root 155 ki31 0K 96K CPU5 5 385:17 99.27% idle{idle: cpu5}
11 root 155 ki31 0K 96K CPU1 1 362:22 54.88% idle{idle: cpu1}
12 root -52 - 0K 224K WAIT 1 161:05 48.19% intr{swi6: task queue}
12 root -88 - 0K 224K WAIT 5 13:57 4.05% intr{irq264: mfi0}
Interrupt rate is also on the higher side in the absence of disk activity.
mfi(4):
root at dell13g-mfi:~ # vmstat -i
interrupt total rate
irq9: acpi0 1 0
irq18: ehci0 ehci1 74619 3
cpu0:timer 779089 32
irq264: mfi0 2645728 110
irq265: bge0 117996 4
cpu2:timer 2862930 119
cpu4:timer 2888148 120
cpu3:timer 2650936 110
cpu1:timer 2567105 106
cpu5:timer 971503 40
Total 15558055 647
Compare the above with mrsas(4):
root at dell13g-mrsas:~ # vmstat -i
interrupt total rate
irq9: acpi0 1 0
irq18: ehci0 ehci1 77600 3
cpu0:timer 557554 22
irq264: mrsas0 10682 0
irq265: mrsas0 8230 0
irq266: mrsas0 8278 0
irq267: mrsas0 8328 0
irq268: mrsas0 8466 0
irq269: mrsas0 8849 0
irq270: bge0 134170 5
cpu4:timer 153721 6
cpu2:timer 145912 6
cpu3:timer 148140 6
cpu1:timer 154170 6
cpu5:timer 108449 4
Total 1532550 63
Procstat output.
root at dell13g-mfi:~ # procstat -kk 12
PID TID COMM TDNAME KSTACK
12 100009 intr swi1: netisr 0 mi_switch+0xe1 ithread_loop+0x190 fork_exit+0x9a fork_trampoline+0xe
12 100010 intr swi4: clock mi_switch+0xe1 ithread_loop+0x190 fork_exit+0x9a fork_trampoline+0xe
12 100011 intr swi4: clock
12 100012 intr swi4: clock
12 100013 intr swi4: clock
12 100014 intr swi4: clock
12 100015 intr swi4: clock
12 100016 intr swi3: vm
12 100023 intr swi6: task queue <running>
12 100024 intr swi6: Giant task mi_switch+0xe1 ithread_loop+0x190 fork_exit+0x9a fork_trampoline+0xe
12 100026 intr swi5: fast taskq
12 100033 intr irq264: mfi0 mi_switch+0xe1 ithread_loop+0x190 fork_exit+0x9a fork_trampoline+0xe
12 100034 intr irq18: ehci0 ehc mi_switch+0xe1 ithread_loop+0x190 fork_exit+0x9a fork_trampoline+0xe
12 100047 intr swi0: uart uart
Dtrace output.
root at dell13g-mfi:~ # kldload dtraceall
root at dell13g-mfi:~ # dtrace -n 'profile:::profile-276hz { @pc[stack()]=count(); }'
dtrace: description 'profile:::profile-276hz ' matched 1 probe
^C
kernel`copyin+0x4b
1
kernel`sched_idletd+0x1bb
1
kernel`spinlock_exit+0x2d
kernel`mfi_tbolt_complete_cmd+0x230
kernel`mfi_intr_tbolt+0x60
kernel`intr_event_execute_handlers+0xab
kernel`ithread_loop+0x96
kernel`fork_exit+0x9a
kernel`0xffffffff80d30d2e
1
kernel`__mtx_lock_sleep+0x210
kernel`fork_exit+0x9a
kernel`0xffffffff80d30d2e
5
kernel`__mtx_lock_sleep+0x221
kernel`fork_exit+0x9a
kernel`0xffffffff80d30d2e
7
kernel`__mtx_lock_sleep+0x21c
kernel`fork_exit+0x9a
kernel`0xffffffff80d30d2e
10
kernel`get_tsc+0x4
kernel`mfi_data_cb+0x24c
kernel`bus_dmamap_load+0xd1
kernel`mfi_mapcmd+0x9a
kernel`mfi_startio+0x3a0
kernel`mfi_wait_command+0x8a
kernel`mfi_tbolt_sync_map_info+0x75
kernel`mfi_handle_map_sync+0x4c
kernel`taskqueue_run_locked+0xe5
kernel`taskqueue_run+0x81
kernel`intr_event_execute_handlers+0xab
kernel`ithread_loop+0x96
kernel`fork_exit+0x9a
kernel`0xffffffff80d30d2e
13
kernel`get_tsc+0x1
kernel`bus_dmamap_load+0xd1
kernel`mfi_mapcmd+0x9a
kernel`mfi_startio+0x3a0
kernel`mfi_wait_command+0x8a
kernel`mfi_tbolt_sync_map_info+0x75
kernel`mfi_handle_map_sync+0x4c
kernel`taskqueue_run_locked+0xe5
kernel`taskqueue_run+0x81
kernel`intr_event_execute_handlers+0xab
kernel`ithread_loop+0x96
kernel`fork_exit+0x9a
kernel`0xffffffff80d30d2e
17
kernel`__mtx_lock_sleep+0x218
kernel`fork_exit+0x9a
kernel`0xffffffff80d30d2e
22
kernel`DELAY+0xd8
kernel`bus_dmamap_load+0xd1
kernel`mfi_mapcmd+0x9a
kernel`mfi_startio+0x3a0
kernel`mfi_wait_command+0x8a
kernel`mfi_tbolt_sync_map_info+0x75
kernel`mfi_handle_map_sync+0x4c
kernel`taskqueue_run_locked+0xe5
kernel`taskqueue_run+0x81
kernel`intr_event_execute_handlers+0xab
kernel`ithread_loop+0x96
kernel`fork_exit+0x9a
kernel`0xffffffff80d30d2e
32
kernel`DELAY+0xd0
kernel`bus_dmamap_load+0xd1
kernel`mfi_mapcmd+0x9a
kernel`mfi_startio+0x3a0
kernel`mfi_wait_command+0x8a
kernel`mfi_tbolt_sync_map_info+0x75
kernel`mfi_handle_map_sync+0x4c
kernel`taskqueue_run_locked+0xe5
kernel`taskqueue_run+0x81
kernel`intr_event_execute_handlers+0xab
kernel`ithread_loop+0x96
kernel`fork_exit+0x9a
kernel`0xffffffff80d30d2e
38
kernel`__mtx_lock_sleep+0x215
kernel`fork_exit+0x9a
kernel`0xffffffff80d30d2e
39
kernel`DELAY+0xf1
kernel`bus_dmamap_load+0xd1
kernel`mfi_mapcmd+0x9a
kernel`mfi_startio+0x3a0
kernel`mfi_wait_command+0x8a
kernel`mfi_tbolt_sync_map_info+0x75
kernel`mfi_handle_map_sync+0x4c
kernel`taskqueue_run_locked+0xe5
kernel`taskqueue_run+0x81
kernel`intr_event_execute_handlers+0xab
kernel`ithread_loop+0x96
kernel`fork_exit+0x9a
kernel`0xffffffff80d30d2e
43
kernel`DELAY+0xe8
kernel`bus_dmamap_load+0xd1
kernel`mfi_mapcmd+0x9a
kernel`mfi_startio+0x3a0
kernel`mfi_wait_command+0x8a
kernel`mfi_tbolt_sync_map_info+0x75
kernel`mfi_handle_map_sync+0x4c
kernel`taskqueue_run_locked+0xe5
kernel`taskqueue_run+0x81
kernel`intr_event_execute_handlers+0xab
kernel`ithread_loop+0x96
kernel`fork_exit+0x9a
kernel`0xffffffff80d30d2e
47
kernel`__mtx_lock_sleep+0x212
kernel`fork_exit+0x9a
kernel`0xffffffff80d30d2e
47
kernel`DELAY+0xd2
kernel`bus_dmamap_load+0xd1
kernel`mfi_mapcmd+0x9a
kernel`mfi_startio+0x3a0
kernel`mfi_wait_command+0x8a
kernel`mfi_tbolt_sync_map_info+0x75
kernel`mfi_handle_map_sync+0x4c
kernel`taskqueue_run_locked+0xe5
kernel`taskqueue_run+0x81
kernel`intr_event_execute_handlers+0xab
kernel`ithread_loop+0x96
kernel`fork_exit+0x9a
kernel`0xffffffff80d30d2e
187
kernel`get_tsc
kernel`bus_dmamap_load+0xd1
kernel`mfi_mapcmd+0x9a
kernel`mfi_startio+0x3a0
kernel`mfi_wait_command+0x8a
kernel`mfi_tbolt_sync_map_info+0x75
kernel`mfi_handle_map_sync+0x4c
kernel`taskqueue_run_locked+0xe5
kernel`taskqueue_run+0x81
kernel`intr_event_execute_handlers+0xab
kernel`ithread_loop+0x96
kernel`fork_exit+0x9a
kernel`0xffffffff80d30d2e
463
kernel`get_tsc+0x6
kernel`mfi_data_cb+0x24c
kernel`bus_dmamap_load+0xd1
kernel`mfi_mapcmd+0x9a
kernel`mfi_startio+0x3a0
kernel`mfi_wait_command+0x8a
kernel`mfi_tbolt_sync_map_info+0x75
kernel`mfi_handle_map_sync+0x4c
kernel`taskqueue_run_locked+0xe5
kernel`taskqueue_run+0x81
kernel`intr_event_execute_handlers+0xab
kernel`ithread_loop+0x96
kernel`fork_exit+0x9a
kernel`0xffffffff80d30d2e
654
kernel`acpi_cpu_c1+0x6
kernel`sched_idletd+0x1d5
kernel`fork_exit+0x9a
kernel`0xffffffff80d30d2e
13283
storcli doesn't work on either of the servers.
root at dell13g-mrsas:~ # storcli show all
Status Code = 0
Status = Success
Description = None
Number of Controllers = 0
Host Name = dell13g-mrsas
Operating System = FreeBSD10.2-RELEASE
root at dell13g-mrsas:~ # storcli64 show all
Status Code = 0
Status = Success
Description = None
Number of Controllers = 0
Host Name = dell13g-mrsas
Operating System = FreeBSD10.2-RELEASE
root at dell13g-mfi:~ # storcli show all
Status Code = 0
Status = Success
Description = None
Number of Controllers = 0
Host Name = dell13g-mfi
Operating System = FreeBSD10.2-RELEASE
root at dell13g-mfi:~ # storcli64 show all
Status Code = 0
Status = Success
Description = None
Number of Controllers = 0
Host Name = dell13g-mfi
Operating System = FreeBSD10.2-RELEASE
Can someone help me debug this problem?
Regards,
Pallav
More information about the freebsd-questions
mailing list