Poor SSD performance with Dell PERC H730 Mini attached to the mfi(4) driver on FreeBSD 10.2

Pallav Bose pallav_bose at yahoo.com
Tue Oct 13 02:45:03 UTC 2015


+freebsd-scsi, Sibananda
MFI_FUSION_ENABLE_INTERRUPT_MASK has been defined differently in mfi(4) vs. mrsas(4) - 0x00000008 vs. 0x00000009. The mrsas(4) #define matches the Linux version. Is there a bug in the way the ISR is being setup in case of mfi(4) + Invader card?

When I print the value of "status" from the function mfi_tbolt_check_clear_intr_ppc(), it equals 0x40000008. Please note that during this time there is no disk activity, but interrupts are continuously being generated (when I check vmstat -i).int32_t
mfi_tbolt_check_clear_intr_ppc(struct mfi_softc *sc)
{
	int32_t status, mfi_status = 0;

	status = MFI_READ4(sc, MFI_OSTS);

	if (status & 1) {
		MFI_WRITE4(sc, MFI_OSTS, status);
		MFI_READ4(sc, MFI_OSTS);
		if (status & MFI_STATE_CHANGE_INTERRUPT) {
			mfi_status |= MFI_FIRMWARE_STATE_CHANGE;
		}

		return mfi_status;
	}
	if (!(status & MFI_FUSION_ENABLE_INTERRUPT_MASK))
		return 1;

	MFI_READ4(sc, MFI_OSTS);
	return 0;
}So, mfi_tbolt_check_clear_intr_ppc() returns 0 to the caller mfi_intr_tbolt() even though the interrupt was a spurious one?
Regards,
Pallav
 


     On Monday, October 5, 2015 3:32 PM, Pallav Bose <pallav_bose at yahoo.com> wrote:
   

 Hi,

Hello,

I have two Dell PowerEdge R430 servers - dell13g-mfi and dell13g-mrsas - both with PERC H730 Mini RAID controllers. I have FreeBSD 10.2 running on each server. One server is using the mfi(4) driver, and the other the mrsas(4) driver. I'm observing very poor SSD throughput on the server using the mfi(4) driver. I saw similarly poor throughput on FreeBSD 8.0 with a patched mfi(4) driver and posted a question (https://lists.freebsd.org/pipermail/freebsd-questions/2015-September/268349.html), so I tried FreeBSD 10.2 and see similar results.

dell13g-mfi info:

root at dell13g-mfi:~ # uname -a
FreeBSD dell13g-mfi 10.2-RELEASE FreeBSD 10.2-RELEASE #0 r286666: Wed Aug 12 15:26:37 UTC 2015     root at releng1.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC  amd64


root at dell13g-mfi:~ # cat /boot/loader.conf
kern.geom.label.gptid.enable="0"
zfs_load="YES"


root at dell13g-mfi:~ # pciconf -bclveV mfi0
mfi0 at pci0:1:0:0:        class=0x010400 card=0x1f491028 chip=0x005d1000 rev=0x02 hdr=0x00
    vendor     = 'LSI Logic / Symbios Logic'
    device     = 'MegaRAID SAS-3 3108 [Invader]'
    class      = mass storage
    subclass   = RAID
    bar   [10] = type I/O Port, range 32, base 0x2000, size 256, enabled
    bar   [14] = type Memory, range 64, base 0x91d00000, size 65536, enabled
    bar   [1c] = type Memory, range 64, base 0x91c00000, size 1048576, enabled
    cap 01[50] = powerspec 3  supports D0 D1 D2 D3  current D0
    cap 10[68] = PCI-Express 2 endpoint max data 256(4096) FLR link x8(x8)
                 speed 8.0(8.0) ASPM disabled(L0s)
    cap 03[d0] = VPD
    cap 05[a8] = MSI supports 1 message, 64 bit, vector masks enabled with 1 message
    cap 11[c0] = MSI-X supports 97 messages
                 Table in map 0x14[0xe000], PBA in map 0x14[0xf000]
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected
    ecap 0019[1e0] = PCIe Sec 1 lane errors 0
    ecap 0004[1c0] = Power Budgeting 1
    ecap 000e[148] = ARI 1
  PCI-e errors = Correctable Error Detected
                 Unsupported Request Detected
     Corrected = Advisory Non-Fatal Error

dell13g-mrsas info:

root at dell13g-mrsas:~ # uname -a
FreeBSD dell13g-mrsas 10.2-RELEASE FreeBSD 10.2-RELEASE #0 r286666: Wed Aug 12 15:26:37 UTC 2015     root at releng1.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC  amd64


root at dell13g-mrsas:~ # cat /boot/loader.conf
kern.geom.label.gptid.enable="0"
zfs_load="YES"
hw.mfi.mrsas_enable=1

root at dell13g-mrsas:~ # pciconf -bclveV mrsas0
mrsas0 at pci0:1:0:0:      class=0x010400 card=0x1f491028 chip=0x005d1000 rev=0x02 hdr=0x00
    vendor     = 'LSI Logic / Symbios Logic'
    device     = 'MegaRAID SAS-3 3108 [Invader]'
    class      = mass storage
    subclass   = RAID
    bar   [10] = type I/O Port, range 32, base 0x2000, size 256, enabled
    bar   [14] = type Memory, range 64, base 0x91d00000, size 65536, enabled
    bar   [1c] = type Memory, range 64, base 0x91c00000, size 1048576, enabled
    cap 01[50] = powerspec 3  supports D0 D1 D2 D3  current D0
    cap 10[68] = PCI-Express 2 endpoint max data 256(4096) FLR link x8(x8)
                 speed 8.0(8.0) ASPM disabled(L0s)
    cap 03[d0] = VPD
    cap 05[a8] = MSI supports 1 message, 64 bit, vector masks
    cap 11[c0] = MSI-X supports 97 messages, enabled
                 Table in map 0x14[0xe000], PBA in map 0x14[0xf000]
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected
    ecap 0019[1e0] = PCIe Sec 1 lane errors 0
    ecap 0004[1c0] = Power Budgeting 1
    ecap 000e[148] = ARI 1
  PCI-e errors = Correctable Error Detected
                 Unsupported Request Detected
     Corrected = Advisory Non-Fatal Error

A simple read from one of the SSDs shows very poor performance in case of mfi(4).

root at dell13g-mfi:~ # dd if=/dev/mfid1 of=/dev/null bs=1m count=100
100+0 records in
100+0 records out
104857600 bytes transferred in 19.956666 secs (5254264 bytes/sec)

Compare that with mrsas(4).

root at dell13g-mrsas:~ # dd if=/dev/da1 of=/dev/null bs=1m count=100
100+0 records in
100+0 records out
104857600 bytes transferred in 0.248799 secs (421455329 bytes/sec)

High CPU usage by swi6 on the server using mfi(4).

root at dell13g-mfi:~ # top -PSHI
last pid:  1779;  load averages:  0.55,  0.50,  0.44                                                                                                                 up 0+06:39:27  14:49:45
244 processes: 7 running, 223 sleeping, 14 waiting
CPU 0:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
CPU 1:  0.0% user,  0.0% nice,  0.0% system, 64.6% interrupt, 35.4% idle
CPU 2:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
CPU 3:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
CPU 4:  0.0% user,  0.0% nice,  0.4% system,  0.0% interrupt, 99.6% idle
CPU 5:  0.0% user,  0.0% nice,  0.0% system,  5.8% interrupt, 94.2% idle
Mem: 3256K Active, 25M Inact, 588M Wired, 30G Free
ARC: 207M Total, 51M MFU, 147M MRU, 16K Anon, 1864K Header, 7317K Other
Swap: 8192M Total, 8192M Free

  PID USERNAME   PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
   11 root       155 ki31     0K    96K CPU0    0 395:01 100.00% idle{idle: cpu0}
   11 root       155 ki31     0K    96K CPU3    3 361:02 100.00% idle{idle: cpu3}
   11 root       155 ki31     0K    96K CPU2    2 357:53 100.00% idle{idle: cpu2}
   11 root       155 ki31     0K    96K RUN     4 358:00  99.85% idle{idle: cpu4}
   11 root       155 ki31     0K    96K CPU5    5 385:17  99.27% idle{idle: cpu5}
   11 root       155 ki31     0K    96K CPU1    1 362:22  54.88% idle{idle: cpu1}
   12 root       -52    -     0K   224K WAIT    1 161:05  48.19% intr{swi6: task queue}
   12 root       -88    -     0K   224K WAIT    5  13:57   4.05% intr{irq264: mfi0}

Interrupt rate is also on the higher side in the absence of disk activity.

mfi(4):

root at dell13g-mfi:~ # vmstat -i
interrupt                          total       rate
irq9: acpi0                            1          0
irq18: ehci0 ehci1                 74619          3
cpu0:timer                        779089         32
irq264: mfi0                     2645728        110
irq265: bge0                      117996          4
cpu2:timer                       2862930        119
cpu4:timer                       2888148        120
cpu3:timer                       2650936        110
cpu1:timer                       2567105        106
cpu5:timer                        971503         40
Total                           15558055        647

Compare the above with mrsas(4):

root at dell13g-mrsas:~ # vmstat -i
interrupt                          total       rate
irq9: acpi0                            1          0
irq18: ehci0 ehci1                 77600          3
cpu0:timer                        557554         22
irq264: mrsas0                     10682          0
irq265: mrsas0                      8230          0
irq266: mrsas0                      8278          0
irq267: mrsas0                      8328          0
irq268: mrsas0                      8466          0
irq269: mrsas0                      8849          0
irq270: bge0                      134170          5
cpu4:timer                        153721          6
cpu2:timer                        145912          6
cpu3:timer                        148140          6
cpu1:timer                        154170          6
cpu5:timer                        108449          4
Total                            1532550         63


Procstat output.

root at dell13g-mfi:~ # procstat -kk 12
  PID    TID COMM             TDNAME           KSTACK
   12 100009 intr             swi1: netisr 0   mi_switch+0xe1 ithread_loop+0x190 fork_exit+0x9a fork_trampoline+0xe
   12 100010 intr             swi4: clock      mi_switch+0xe1 ithread_loop+0x190 fork_exit+0x9a fork_trampoline+0xe
   12 100011 intr             swi4: clock
   12 100012 intr             swi4: clock
   12 100013 intr             swi4: clock
   12 100014 intr             swi4: clock
   12 100015 intr             swi4: clock
   12 100016 intr             swi3: vm
   12 100023 intr             swi6: task queue <running>
   12 100024 intr             swi6: Giant task mi_switch+0xe1 ithread_loop+0x190 fork_exit+0x9a fork_trampoline+0xe
   12 100026 intr             swi5: fast taskq
   12 100033 intr             irq264: mfi0     mi_switch+0xe1 ithread_loop+0x190 fork_exit+0x9a fork_trampoline+0xe
   12 100034 intr             irq18: ehci0 ehc mi_switch+0xe1 ithread_loop+0x190 fork_exit+0x9a fork_trampoline+0xe
   12 100047 intr             swi0: uart uart

Dtrace output.

root at dell13g-mfi:~ # kldload dtraceall
root at dell13g-mfi:~ # dtrace -n 'profile:::profile-276hz { @pc[stack()]=count(); }'
dtrace: description 'profile:::profile-276hz ' matched 1 probe
^C


              kernel`copyin+0x4b
                1

              kernel`sched_idletd+0x1bb
                1

              kernel`spinlock_exit+0x2d
              kernel`mfi_tbolt_complete_cmd+0x230
              kernel`mfi_intr_tbolt+0x60
              kernel`intr_event_execute_handlers+0xab
              kernel`ithread_loop+0x96
              kernel`fork_exit+0x9a
              kernel`0xffffffff80d30d2e
                1

              kernel`__mtx_lock_sleep+0x210
              kernel`fork_exit+0x9a
              kernel`0xffffffff80d30d2e
                5

              kernel`__mtx_lock_sleep+0x221
              kernel`fork_exit+0x9a
              kernel`0xffffffff80d30d2e
                7

              kernel`__mtx_lock_sleep+0x21c
              kernel`fork_exit+0x9a
              kernel`0xffffffff80d30d2e
               10

              kernel`get_tsc+0x4
              kernel`mfi_data_cb+0x24c
              kernel`bus_dmamap_load+0xd1
              kernel`mfi_mapcmd+0x9a
              kernel`mfi_startio+0x3a0
              kernel`mfi_wait_command+0x8a
              kernel`mfi_tbolt_sync_map_info+0x75
              kernel`mfi_handle_map_sync+0x4c
              kernel`taskqueue_run_locked+0xe5
              kernel`taskqueue_run+0x81
              kernel`intr_event_execute_handlers+0xab
              kernel`ithread_loop+0x96
              kernel`fork_exit+0x9a
              kernel`0xffffffff80d30d2e
               13

              kernel`get_tsc+0x1
              kernel`bus_dmamap_load+0xd1
              kernel`mfi_mapcmd+0x9a
              kernel`mfi_startio+0x3a0
              kernel`mfi_wait_command+0x8a
              kernel`mfi_tbolt_sync_map_info+0x75
              kernel`mfi_handle_map_sync+0x4c
              kernel`taskqueue_run_locked+0xe5
              kernel`taskqueue_run+0x81
              kernel`intr_event_execute_handlers+0xab
              kernel`ithread_loop+0x96
              kernel`fork_exit+0x9a
              kernel`0xffffffff80d30d2e
               17

              kernel`__mtx_lock_sleep+0x218
              kernel`fork_exit+0x9a
              kernel`0xffffffff80d30d2e
               22

              kernel`DELAY+0xd8
              kernel`bus_dmamap_load+0xd1
              kernel`mfi_mapcmd+0x9a
              kernel`mfi_startio+0x3a0
              kernel`mfi_wait_command+0x8a
              kernel`mfi_tbolt_sync_map_info+0x75
              kernel`mfi_handle_map_sync+0x4c
              kernel`taskqueue_run_locked+0xe5
              kernel`taskqueue_run+0x81
              kernel`intr_event_execute_handlers+0xab
              kernel`ithread_loop+0x96
              kernel`fork_exit+0x9a
              kernel`0xffffffff80d30d2e
               32

              kernel`DELAY+0xd0
              kernel`bus_dmamap_load+0xd1
              kernel`mfi_mapcmd+0x9a
              kernel`mfi_startio+0x3a0
              kernel`mfi_wait_command+0x8a
              kernel`mfi_tbolt_sync_map_info+0x75
              kernel`mfi_handle_map_sync+0x4c
              kernel`taskqueue_run_locked+0xe5
              kernel`taskqueue_run+0x81
              kernel`intr_event_execute_handlers+0xab
              kernel`ithread_loop+0x96
              kernel`fork_exit+0x9a
              kernel`0xffffffff80d30d2e
               38

              kernel`__mtx_lock_sleep+0x215
              kernel`fork_exit+0x9a
              kernel`0xffffffff80d30d2e
               39

              kernel`DELAY+0xf1
              kernel`bus_dmamap_load+0xd1
              kernel`mfi_mapcmd+0x9a
              kernel`mfi_startio+0x3a0
              kernel`mfi_wait_command+0x8a
              kernel`mfi_tbolt_sync_map_info+0x75
              kernel`mfi_handle_map_sync+0x4c
              kernel`taskqueue_run_locked+0xe5
              kernel`taskqueue_run+0x81
              kernel`intr_event_execute_handlers+0xab
              kernel`ithread_loop+0x96
              kernel`fork_exit+0x9a
              kernel`0xffffffff80d30d2e
               43

              kernel`DELAY+0xe8
              kernel`bus_dmamap_load+0xd1
              kernel`mfi_mapcmd+0x9a
              kernel`mfi_startio+0x3a0
              kernel`mfi_wait_command+0x8a
              kernel`mfi_tbolt_sync_map_info+0x75
              kernel`mfi_handle_map_sync+0x4c
              kernel`taskqueue_run_locked+0xe5
              kernel`taskqueue_run+0x81
              kernel`intr_event_execute_handlers+0xab
              kernel`ithread_loop+0x96
              kernel`fork_exit+0x9a
              kernel`0xffffffff80d30d2e
               47

              kernel`__mtx_lock_sleep+0x212
              kernel`fork_exit+0x9a
              kernel`0xffffffff80d30d2e
               47

              kernel`DELAY+0xd2
              kernel`bus_dmamap_load+0xd1
              kernel`mfi_mapcmd+0x9a
              kernel`mfi_startio+0x3a0
              kernel`mfi_wait_command+0x8a
              kernel`mfi_tbolt_sync_map_info+0x75
              kernel`mfi_handle_map_sync+0x4c
              kernel`taskqueue_run_locked+0xe5
              kernel`taskqueue_run+0x81
              kernel`intr_event_execute_handlers+0xab
              kernel`ithread_loop+0x96
              kernel`fork_exit+0x9a
              kernel`0xffffffff80d30d2e
              187

              kernel`get_tsc
              kernel`bus_dmamap_load+0xd1
              kernel`mfi_mapcmd+0x9a
              kernel`mfi_startio+0x3a0
              kernel`mfi_wait_command+0x8a
              kernel`mfi_tbolt_sync_map_info+0x75
              kernel`mfi_handle_map_sync+0x4c
              kernel`taskqueue_run_locked+0xe5
              kernel`taskqueue_run+0x81
              kernel`intr_event_execute_handlers+0xab
              kernel`ithread_loop+0x96
              kernel`fork_exit+0x9a
              kernel`0xffffffff80d30d2e
              463

              kernel`get_tsc+0x6
              kernel`mfi_data_cb+0x24c
              kernel`bus_dmamap_load+0xd1
              kernel`mfi_mapcmd+0x9a
              kernel`mfi_startio+0x3a0
              kernel`mfi_wait_command+0x8a
              kernel`mfi_tbolt_sync_map_info+0x75
              kernel`mfi_handle_map_sync+0x4c
              kernel`taskqueue_run_locked+0xe5
              kernel`taskqueue_run+0x81
              kernel`intr_event_execute_handlers+0xab
              kernel`ithread_loop+0x96
              kernel`fork_exit+0x9a
              kernel`0xffffffff80d30d2e
              654

              kernel`acpi_cpu_c1+0x6
              kernel`sched_idletd+0x1d5
              kernel`fork_exit+0x9a
              kernel`0xffffffff80d30d2e
            13283

storcli doesn't work on either of the servers.

root at dell13g-mrsas:~ # storcli show all
Status Code = 0
Status = Success
Description = None

Number of Controllers = 0
Host Name = dell13g-mrsas
Operating System  = FreeBSD10.2-RELEASE

root at dell13g-mrsas:~ # storcli64 show all
Status Code = 0
Status = Success
Description = None

Number of Controllers = 0
Host Name = dell13g-mrsas
Operating System  = FreeBSD10.2-RELEASE

root at dell13g-mfi:~ # storcli show all
Status Code = 0
Status = Success
Description = None

Number of Controllers = 0
Host Name = dell13g-mfi
Operating System  = FreeBSD10.2-RELEASE

root at dell13g-mfi:~ # storcli64 show all
Status Code = 0
Status = Success
Description = None

Number of Controllers = 0
Host Name = dell13g-mfi
Operating System  = FreeBSD10.2-RELEASE

Can someone help me debug this problem?
 
Regards,
Pallav




  


More information about the freebsd-scsi mailing list