[Bug 243063] [bhyve] nvme timeouts

[bugzilla-noreply at freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=243063

Mateusz Kwiatkowski <kwiat3k at panic.pl> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |kwiat3k at panic.pl

--- Comment #1 from Mateusz Kwiatkowski <kwiat3k at panic.pl> ---
I also performed some bhyve+nvme tests.
Host: FreeBSD 13.0 r356983, Opteron 6128, ZFS raid10 on 4 HDDs
Guests: FreeBSD 12.1-RELEASE

bhyveload -c /dev/nmdm-1_hv-3.1A -m 2048M -e autoboot_delay=3 -d
/dev/zvol/zroot/vm/1_hv-3/disk0 1_hv-3
  [bhyve options: -c 2 -m 2048M -AHP -U 3805a422-c9c8-46c5-aa8d-31142e90ea89
-u]
  [bhyve devices: -s 0,hostbridge -s 31,lpc -s
4:0,nvme,/dev/zvol/zroot/vm/1_hv-3/disk0 -s
4:1,ahci-cd,/zroot/vm/1_hv-3/seed.iso -s
5:0,virtio-net,tap2,mac=58:9c:fc:0f:3e:27]
  [bhyve console: -l com1,/dev/nmdm-1_hv-3.1A]

# zfs get volblocksize zroot/vm/1_hv-3/disk0
NAME                   PROPERTY      VALUE     SOURCE
zroot/vm/1_hv-3/disk0  volblocksize  8K        default


Performance comparison of virtio-blk/ahci-hd/nvme:

Test command: fio --name=test --iodepth=4 --rw=randrw:2 --rwmixread=70
--rwmixwrite=30 --bs=8k --direct=0 --size=256m --numjobs=8

baremetal:
   READ: bw=456MiB/s (478MB/s), 57.3MiB/s-65.5MiB/s (60.1MB/s-68.6MB/s),
io=1433MiB (1502MB), run=2719-3142msec
  WRITE: bw=196MiB/s (205MB/s), 24.1MiB/s-28.7MiB/s (25.3MB/s-30.1MB/s),
io=615MiB (645MB), run=2719-3142msec

virtio-blk:
   READ: bw=115MiB/s (121MB/s), 14.4MiB/s-20.5MiB/s (15.1MB/s-21.5MB/s),
io=1433MiB (1502MB), run=8773-12453msec
  WRITE: bw=49.4MiB/s (51.8MB/s), 6348KiB/s-8851KiB/s (6501kB/s-9063kB/s),
io=615MiB (645MB), run=8773-12453msec

ahci-hd:
   READ: bw=110MiB/s (116MB/s), 13.8MiB/s-36.4MiB/s (14.4MB/s-38.1MB/s),
io=1433MiB (1502MB), run=4940-12968msec
  WRITE: bw=47.4MiB/s (49.7MB/s), 6107KiB/s-15.5MiB/s (6254kB/s-16.2MB/s),
io=615MiB (645MB), run=4940-12968msec

nvme:
   READ: bw=19.7MiB/s (20.7MB/s), 2520KiB/s-4811KiB/s (2580kB/s-4926kB/s),
io=1433MiB (1502MB), run=38351-72662msec
  WRITE: bw=8671KiB/s (8879kB/s), 1083KiB/s-2025KiB/s (1109kB/s-2073kB/s),
io=615MiB (645MB), run=38351-72662msec


>From guest's log:

nvme0: Missing interrupt
nvme0: Missing interrupt
nvme0: nvme0: cpl does not map to outstanding cmd
Missing interrupt
cdw0:00000000 sqhd:0019 sqid:0002 cid:0066 p:1 sc:00 sct:0 m:0 dnr:0
nvme0: Resetting controller due to a timeout.
nvme0: resetting controller
nvme0: temperature threshold not supported
nvme0: aborting outstanding i/o
nvme0: resubmitting queued i/o
nvme0: WRITE sqid:2 cid:0 nsid:1 lba:8752079 len:8

-- 
You are receiving this mail because:
You are the assignee for the bug.