Question about bottle neck in storage

[Warner Losh]

On Tue, Sep 24, 2019 at 5:46 PM John Fleming <john at spikefishsolutions.com>
wrote:

> Is there anyway to see how busy a SAS/Sata controller is vs disks? I
> have a R720 with 14 Samsung 860 EVOs in it (its a lab server) in raid
> 10 ZFS.
>
> When firing off a dd I (bs=1G count=10) seems like the disks never go
> above %50 busy. I'm trying to figure out if i'm maxing out SATA 3 BW
> or if its something else (like terrible dd options).
>

Two points to consider here. First, NVMe has lots of queues and needs lots
of concurrent transactions to saturate, so the 50% busy means you are no
where close to saturating the drives. Schedule more I/O too fix that. It's
better to do lots and lots of concurrent DD to different parts of the
drive, or to use fio with the aio kernel option and posixaio I/O scheduling
method.

I use the following script, but often need to increase the number of
threads / jobs to saturate.

; SSD testing: 128k I/O 64 jobs 32 deep queue

[global]
direct=1
rw=randread
refill_buffers
norandommap
randrepeat=0
bs=128k
ioengine=posixaio
iodepth=32
numjobs=64
runtime=60
group_reporting
thread

[ssd128k]

Second, the system's % busy statistics are misleading. They are the %of the
time that a command is outstanding on the drive. 100% busy can be a tiny
percentage of the total bandwidth you can get from the drive.


> my setup is Dell R720 with 2 x LSI 9361 cards. Each card is going to a
> dedicated 8 drive board inside the front of the R720. Basically i'm
> just saying its not a single SAS cable to 14 drives.
>
> Don't have cpu info hand.. zeon something. DDR3-1600 (128GB)
>
> Both controllers are in 8x slots running PCIe gen 3.
>
> BTW i'm sure this has been asked a million times but what would be
> some decent benchmark tests while i'm at it?
>

See above... :)

Warner