Re: bhyve NVMe 1.4 support

From: Bakul Shah <bakul_at_iitbombay.org>
Date: Sun, 20 Mar 2022 22:34:15 UTC 
On Mar 20, 2022, at 3:11 PM, Mario Marietto <marietto2008@gmail.com> wrote:
> 
> -s 2,nvme,/dev/nvd0 \

AIUI, this 'nvme' means bhyve will *emulate* an NVME device
and it will treat whatever is given to it (/dev/nvd0) as just
dumb storage. It doesn't care if /dev/nvd0 is an NVME device.
So commands such as nvmecontrol identify etc. in the *guest*
won't be passed through to /dev/nvd0 through to it. If you
run nvmecontrol identify under ktrace in the guest, you will
see something like

  3657 nvmecontrol NAMI  "/dev/nvme0ns1"
  3657 nvmecontrol RET   openat 3
  3657 nvmecontrol CALL  ioctl(0x3,NVME_GET_NSID,0x1245f31731a8)
  3657 nvmecontrol RET   ioctl 0
  3657 nvmecontrol CALL  close(0x3)
  3657 nvmecontrol RET   close 0
  3657 nvmecontrol CALL  openat(AT_FDCWD,0x1245f3173260,0<O_RDONLY>)
  3657 nvmecontrol NAMI  "/dev/nvme0"
  3657 nvmecontrol RET   openat 3
  3657 nvmecontrol CALL  ioctl(0x3,NVME_PASSTHROUGH_CMD,0x1245f3173250)
  3657 nvmecontrol RET   ioctl 0
  3657 nvmecontrol CALL  close(0x3)

These ioctls will *not* be passed on the real /dev/nvd0 in the host
so they are *not* going to return the same information.

If you want to talk to the real nvme device from a guest, you have
to use a passthrough device. Though I don't know how well this works
(or if at all).