From nobody Sun Mar 20 19:02:41 2022 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 9DF481A10C8A for ; Sun, 20 Mar 2022 19:03:18 +0000 (UTC) (envelope-from marietto2008@gmail.com) Received: from mail-yw1-x1132.google.com (mail-yw1-x1132.google.com [IPv6:2607:f8b0:4864:20::1132]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KM6bB0gLhz4ly1 for ; Sun, 20 Mar 2022 19:03:18 +0000 (UTC) (envelope-from marietto2008@gmail.com) Received: by mail-yw1-x1132.google.com with SMTP id 00721157ae682-2e5e31c34bfso56739427b3.10 for ; Sun, 20 Mar 2022 12:03:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=i8VCEDeeezAzdq8hhZmVQFdW3dWwdrwcSQSEW6rkIsI=; b=duKopke1kU3AXFQh7Srgi6QTcjulzv7BgQY4Bl6a6OyNma9BcUAeeP5r/r2E4U4hKL zIEgon6QTCAWEelXBL/VT2LZcPC07FPjQqqysC008NEk+XlmJib+qDhtdmjMH0ymfYl3 Isrgzrlu7B/TNggBU9LWObGVxLLy9eN3RxergUDSW/M8bLtsS3n6q2vtztX1Qf8vOji1 z6TEvhPrNV0uOdkjcfe1AF/QOirW+9rK2VAbZpMkBfpVf2sD9XTRPbXa8AMtGa/NaYv4 q4FPWqMb/JDfSFivWe5hMoDZM6yAJ8Z4whUZ+/5snaqeyWC+wKJxHp41DAEFhtbEJ//4 xQJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=i8VCEDeeezAzdq8hhZmVQFdW3dWwdrwcSQSEW6rkIsI=; b=vku7K7u85MwTMMOLM7khtaDyXMYNSb9ZR3Om0Uoil/6TDDjLAsRuZucurMWeJwbwhv 1BZXkrcMM3zwhTHNtvnWPx6+qbreRwyUTBVMbR+DvwT7lj17/72hPpuCOg6wlFFDbzoJ cNlO4pFHcQ5rs5TN6qMNbUa2YbNLBcWrE1a4VeiKHQc7VMCZbLN5fuO0lT8f1dYi/DVn V0w/RdmsZ/J4aunv0EbzNPHirhvMprQaQvZFfKJftpCBRscx4mKAUQ+P8LDlfVbpMGKf yp5zJboU9IA5tfs7IOe1KIJ7vAAOYMbI5Ai8Kvf8ftTUgZeuUHWfSwCUxAkkupRkGwX0 euCw== X-Gm-Message-State: AOAM530sQ/neQVlPcK3eUKo4Nno5/7G1JC5IL/u9zA+ar2yOxzflOvOC GUgX4xjUZCtQYyERxOJOscydcHgvzlMgZ34MNxI= X-Google-Smtp-Source: ABdhPJzcanohHe3WI78TXp6eBt3+GD9cEnSWjNdAAuJgw4ge8vgdy2l0zb+C5gtuFwF3Rgw1rvxCwQR0N5XRQiESONs= X-Received: by 2002:a81:db4a:0:b0:2e5:e999:29d3 with SMTP id b10-20020a81db4a000000b002e5e99929d3mr10228601ywn.397.1647802997460; Sun, 20 Mar 2022 12:03:17 -0700 (PDT) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 References: <00bf01d80104$e6ba5de0$b42f19a0$@tubnor.net> <082b01d80697$64e95030$2ebbf090$@tubnor.net> In-Reply-To: From: Mario Marietto Date: Sun, 20 Mar 2022 20:02:41 +0100 Message-ID: Subject: Re: bhyve NVMe 1.4 support To: Chuck Tuffli Cc: jason@tubnor.net, FreeBSD virtualization Content-Type: multipart/alternative; boundary="00000000000052d23605daab0835" X-Rspamd-Queue-Id: 4KM6bB0gLhz4ly1 X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=duKopke1; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of marietto2008@gmail.com designates 2607:f8b0:4864:20::1132 as permitted sender) smtp.mailfrom=marietto2008@gmail.com X-Spamd-Result: default: False [0.00 / 15.00]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; NEURAL_HAM_LONG(-1.00)[-0.999]; TO_DN_SOME(0.00)[]; NEURAL_SPAM_MEDIUM(1.00)[1.000]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::1132:from]; NEURAL_SPAM_SHORT(1.00)[0.999]; MLMMJ_DEST(0.00)[freebsd-virtualization]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-ThisMailContainsUnwantedMimeParts: N --00000000000052d23605daab0835 Content-Type: text/plain; charset="UTF-8" on FreeBSD : root@marietto:/usr/home/marietto/bhyve # hd -n 256 /dev/nvd0 00000000 33 c0 8e d0 bc 00 7c 8e c0 8e d8 be 00 7c bf 00 |3.....|......|..| 00000010 06 b9 00 02 fc f3 a4 50 68 1c 06 cb fb b9 04 00 |.......Ph.......| 00000020 bd be 07 80 7e 00 00 7c 0b 0f 85 0e 01 83 c5 10 |....~..|........| 00000030 e2 f1 cd 18 88 56 00 55 c6 46 11 05 c6 46 10 00 |.....V.U.F...F..| 00000040 b4 41 bb aa 55 cd 13 5d 72 0f 81 fb 55 aa 75 09 |.A..U..]r...U.u.| 00000050 f7 c1 01 00 74 03 fe 46 10 66 60 80 7e 10 00 74 |....t..F.f`.~..t| 00000060 26 66 68 00 00 00 00 66 ff 76 08 68 00 00 68 00 |&fh....f.v.h..h.| 00000070 7c 68 01 00 68 10 00 b4 42 8a 56 00 8b f4 cd 13 ||h..h...B.V.....| 00000080 9f 83 c4 10 9e eb 14 b8 01 02 bb 00 7c 8a 56 00 |............|.V.| 00000090 8a 76 01 8a 4e 02 8a 6e 03 cd 13 66 61 73 1c fe |.v..N..n...fas..| 000000a0 4e 11 75 0c 80 7e 00 80 0f 84 8a 00 b2 80 eb 84 |N.u..~..........| 000000b0 55 32 e4 8a 56 00 cd 13 5d eb 9e 81 3e fe 7d 55 |U2..V...]...>.}U| 000000c0 aa 75 6e ff 76 00 e8 8d 00 75 17 fa b0 d1 e6 64 |.un.v....u.....d| 000000d0 e8 83 00 b0 df e6 60 e8 7c 00 b0 ff e6 64 e8 75 |......`.|....d.u| 000000e0 00 fb b8 00 bb cd 1a 66 23 c0 75 3b 66 81 fb 54 |.......f#.u;f..T| 000000f0 43 50 41 75 32 81 f9 02 01 72 2c 66 68 07 bb 00 |CPAu2....r,fh...| 00000100 On Ubuntu 21.10 : root@marietto-BHYVE:/home/marietto# hd -n 256 /dev/nvme0n1 00000000 33 c0 8e d0 bc 00 7c 8e c0 8e d8 be 00 7c bf 00 |3.....|......|..| 00000010 06 b9 00 02 fc f3 a4 50 68 1c 06 cb fb b9 04 00 |.......Ph.......| 00000020 bd be 07 80 7e 00 00 7c 0b 0f 85 0e 01 83 c5 10 |....~..|........| 00000030 e2 f1 cd 18 88 56 00 55 c6 46 11 05 c6 46 10 00 |.....V.U.F...F..| 00000040 b4 41 bb aa 55 cd 13 5d 72 0f 81 fb 55 aa 75 09 |.A..U..]r...U.u.| 00000050 f7 c1 01 00 74 03 fe 46 10 66 60 80 7e 10 00 74 |....t..F.f`.~..t| 00000060 26 66 68 00 00 00 00 66 ff 76 08 68 00 00 68 00 |&fh....f.v.h..h.| 00000070 7c 68 01 00 68 10 00 b4 42 8a 56 00 8b f4 cd 13 ||h..h...B.V.....| 00000080 9f 83 c4 10 9e eb 14 b8 01 02 bb 00 7c 8a 56 00 |............|.V.| 00000090 8a 76 01 8a 4e 02 8a 6e 03 cd 13 66 61 73 1c fe |.v..N..n...fas..| 000000a0 4e 11 75 0c 80 7e 00 80 0f 84 8a 00 b2 80 eb 84 |N.u..~..........| 000000b0 55 32 e4 8a 56 00 cd 13 5d eb 9e 81 3e fe 7d 55 |U2..V...]...>.}U| 000000c0 aa 75 6e ff 76 00 e8 8d 00 75 17 fa b0 d1 e6 64 |.un.v....u.....d| 000000d0 e8 83 00 b0 df e6 60 e8 7c 00 b0 ff e6 64 e8 75 |......`.|....d.u| 000000e0 00 fb b8 00 bb cd 1a 66 23 c0 75 3b 66 81 fb 54 |.......f#.u;f..T| 000000f0 43 50 41 75 32 81 f9 02 01 72 2c 66 68 07 bb 00 |CPAu2....r,fh...| 00000100 Il giorno dom 20 mar 2022 alle ore 19:35 Chuck Tuffli ha scritto: > On Sun, Mar 20, 2022 at 8:13 AM Mario Marietto > wrote: > > > > ----> Help me understand what "not recognized" means. The device nvme0n1 > : > > > > I don't see the partitions that are stored inside the disk nvme. And I'm > not able to mount the NTFS partition that's mapped as nvd0p2 under FreeBSD. > > Thank you, I understand the question now. As an experiment, I created > a zvol, copied a FreeBSD disk image to it, and verified that fdisk > showed what I expected on the guest. I.e. : > > # zfs create -V 20G zroot/vmvol/gptdisk > # dd if=/vms/.img/FreeBSD-14.0-CURRENT-amd64.raw > of=/dev/zvol/zroot/vmvol/gptdisk bs=1m > # gpart recover zvol/zroot/vmvol/gptdisk > < add /dev/zvol/zroot/vmvol/gptdisk to test-vm configuration > > # vm start test-vm > # ssh root@test-vm lsb_release -a > Distributor ID: Ubuntu > Description: Ubuntu 21.04 > Release: 21.04 > Codename: hirsute > No LSB modules are available. > # ssh root@test-vm fdisk -l /dev/nvme1n1 > Disk /dev/nvme1n1: 20 GiB, 21474836480 bytes, 41943040 sectors > Disk model: bhyve-NVMe > Units: sectors of 1 * 512 = 512 bytes > Sector size (logical/physical): 512 bytes / 512 bytes > I/O size (minimum/optimal): 512 bytes / 512 bytes > Disklabel type: gpt > Disk identifier: 1B73327C-EAE2-11EB-90A0-002590EC5BF2 > > Device Start End Sectors Size Type > /dev/nvme1n1p1 3 129 127 63.5K FreeBSD boot > /dev/nvme1n1p2 130 66713 66584 32.5M EFI System > /dev/nvme1n1p3 66714 2163865 2097152 1G FreeBSD swap > /dev/nvme1n1p4 2163866 10552473 8388608 4G FreeBSD UFS > > The bhyve invocation is: > bhyve -c 2 -m 2G -Hw \ > -s 0,hostbridge \ > -s 4:0,virtio-blk,/dev/zvol/zroot/vms/test-vm/disk0 \ > -s 5:0,nvme,/dev/zvol/zroot/vmvol/disk0 \ > -s 6:0,nvme,/dev/zvol/zroot/vmvol/gptdisk \ > -s 7:0,virtio-net,tap0,mac=58:9c:fc:0b:ed:d6 > -s 31,lpc \ > -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \ > -l com1,/dev/nmdm-test-vm.1A \ > test-vm > > Note that my experiment is different from what you are doing. But I > would expect a Zvol and raw block device to behave the same as a > backing-store for an emulated NVMe drive in bhyve. My experiment ran > on -current, but the behavior in this area should be identical to the > 13.0-p8 version you are using. > > Does the output on the host of > # hd -n 256 /dev/nvd0 > match the output on the guest of > # hd -n 256 /dev/nvme0n1 > ? > -- Mario. --00000000000052d23605daab0835 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
on FreeBSD :

root@marietto:/usr/home/marietto/bhyve # hd -n 256 /dev/nvd0

00000000 =C2=A033 c0 8e= d0 bc 00 7c 8e =C2=A0c0 8e d8 be 00 7c bf 00 =C2=A0|3.....|......|..|
00000010 =C2=A006 b9 00 02 fc f3 a4 50 =C2=A068 1c 06 cb fb b9 04 00 = =C2=A0|.......Ph.......|
00000020 =C2=A0bd be 07 80 7e 00 00 7c =C2=A00b 0f 85 0e 01 83 c5 10 = =C2=A0|....~..|........|
00000030 =C2=A0e2 f1 cd 18 88 56 00 55 =C2=A0c6 46 11 05 c6 46 10 00 = =C2=A0|.....V.U.F...F..|
00000040 =C2=A0b4 41 bb aa 55 cd 13 5d =C2=A072 0f 81 fb 55 aa 75 09 = =C2=A0|.A..U..]r...U.u.|
00000050 =C2=A0f7 c1 01 00 74 03 fe 46 =C2=A010 66 60 80 7e 10 00 74 = =C2=A0|....t..F.f`.~..t|
00000060 =C2=A026 66 68 00 00 00 00 66 =C2=A0ff 76 08 68 00 00 68 00 = =C2=A0|&fh....f.v.h..h.|
00000070 =C2=A07c 68 01 00 68 10 00 b4 =C2=A042 8a 56 00 8b f4 cd 13 = =C2=A0||h..h...B.V.....|
00000080 =C2=A09f 83 c4 10 9e eb 14 b8 =C2=A001 02 bb 00 7c 8a 56 00 = =C2=A0|............|.V.|
00000090 =C2=A08a 76 01 8a 4e 02 8a 6e =C2=A003 cd 13 66 61 73 1c fe = =C2=A0|.v..N..n...fas..|
000000a0 =C2=A04e 11 75 0c 80 7e 00 80 =C2=A00f 84 8a 00 b2 80 eb 84 = =C2=A0|N.u..~..........|
000000b0 =C2=A055 32 e4 8a 56 00 cd 13 =C2=A05d eb 9e 81 3e fe 7d 55 = =C2=A0|U2..V...]...>.}U|
000000c0 =C2=A0aa 75 6e ff 76 00 e8 8d =C2=A000 75 17 fa b0 d1 e6 64 = =C2=A0|.un.v....u.....d|
000000d0 =C2=A0e8 83 00 b0 df e6 60 e8 =C2=A07c 00 b0 ff e6 64 e8 75 = =C2=A0|......`.|....d.u|
000000e0 =C2=A000 fb b8 00 bb cd 1a 66 =C2=A023 c0 75 3b 66 81 fb 54 = =C2=A0|.......f#.u;f..T|
000000f0 =C2=A043 50 41 75 32 81 f9 02 =C2=A001 72 2c 66 68 07 bb 00 = =C2=A0|CPAu2....r,fh...|
00000100

On Ubuntu 21.10 :

root@marietto-BHYVE:/home/marietto# hd -n 256 /dev/= nvme0n1

00000000 =C2=A033 c0 8e= d0 bc 00 7c 8e =C2=A0c0 8e d8 be 00 7c bf 00 =C2=A0|3.....|......|..|
0= 0000010 =C2=A006 b9 00 02 fc f3 a4 50 =C2=A068 1c 06 cb fb b9 04 00 =C2=A0|= .......Ph.......|
00000020 =C2=A0bd be 07 80 7e 00 00 7c =C2=A00b 0f 85 = 0e 01 83 c5 10 =C2=A0|....~..|........|
00000030 =C2=A0e2 f1 cd 18 88 56= 00 55 =C2=A0c6 46 11 05 c6 46 10 00 =C2=A0|.....V.U.F...F..|
00000040 = =C2=A0b4 41 bb aa 55 cd 13 5d =C2=A072 0f 81 fb 55 aa 75 09 =C2=A0|.A..U..]= r...U.u.|
00000050 =C2=A0f7 c1 01 00 74 03 fe 46 =C2=A010 66 60 80 7e 10= 00 74 =C2=A0|....t..F.f`.~..t|
00000060 =C2=A026 66 68 00 00 00 00 66 = =C2=A0ff 76 08 68 00 00 68 00 =C2=A0|&fh....f.v.h..h.|
00000070 =C2= =A07c 68 01 00 68 10 00 b4 =C2=A042 8a 56 00 8b f4 cd 13 =C2=A0||h..h...B.V= .....|
00000080 =C2=A09f 83 c4 10 9e eb 14 b8 =C2=A001 02 bb 00 7c 8a 56= 00 =C2=A0|............|.V.|
00000090 =C2=A08a 76 01 8a 4e 02 8a 6e =C2= =A003 cd 13 66 61 73 1c fe =C2=A0|.v..N..n...fas..|
000000a0 =C2=A04e 11= 75 0c 80 7e 00 80 =C2=A00f 84 8a 00 b2 80 eb 84 =C2=A0|N.u..~..........|000000b0 =C2=A055 32 e4 8a 56 00 cd 13 =C2=A05d eb 9e 81 3e fe 7d 55 =C2= =A0|U2..V...]...>.}U|
000000c0 =C2=A0aa 75 6e ff 76 00 e8 8d =C2=A000= 75 17 fa b0 d1 e6 64 =C2=A0|.un.v....u.....d|
000000d0 =C2=A0e8 83 00 b= 0 df e6 60 e8 =C2=A07c 00 b0 ff e6 64 e8 75 =C2=A0|......`.|....d.u|
000= 000e0 =C2=A000 fb b8 00 bb cd 1a 66 =C2=A023 c0 75 3b 66 81 fb 54 =C2=A0|..= .....f#.u;f..T|
000000f0 =C2=A043 50 41 75 32 81 f9 02 =C2=A001 72 2c 66= 68 07 bb 00 =C2=A0|CPAu2....r,fh...|
00000100


Il giorno dom 20 mar 2022 al= le ore 19:35 Chuck Tuffli <chuck@tuf= fli.net> ha scritto:
On Sun, Mar 20, 2022 at 8:13 AM Mario Marietto <marietto2008@gmail.com&= gt; wrote:
>
> ----> Help me understand what "not recognized" means. The= device nvme0n1 :
>
> I don't see the partitions that are stored inside the disk nvme. A= nd I'm not able to mount the NTFS partition that's mapped as nvd0p2= under FreeBSD.

Thank you, I understand the question now. As an experiment, I created
a zvol, copied a FreeBSD disk image to it, and verified that fdisk
showed what I expected on the guest. I.e. :

#=C2=A0 zfs create -V 20G zroot/vmvol/gptdisk
#=C2=A0 dd if=3D/vms/.img/FreeBSD-14.0-CURRENT-amd64.raw
of=3D/dev/zvol/zroot/vmvol/gptdisk bs=3D1m
#=C2=A0 gpart recover zvol/zroot/vmvol/gptdisk
< add /dev/zvol/zroot/vmvol/gptdisk to test-vm configuration >
# vm start test-vm
# ssh root@test-vm lsb_release -a
Distributor ID: Ubuntu
Description:=C2=A0 =C2=A0 Ubuntu 21.04
Release:=C2=A0 =C2=A0 =C2=A0 =C2=A0 21.04
Codename:=C2=A0 =C2=A0 =C2=A0 =C2=A0hirsute
No LSB modules are available.
#=C2=A0 ssh root@test-vm fdisk -l /dev/nvme1n1
Disk /dev/nvme1n1: 20 GiB, 21474836480 bytes, 41943040 sectors
Disk model: bhyve-NVMe
Units: sectors of 1 * 512 =3D 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 1B73327C-EAE2-11EB-90A0-002590EC5BF2

Device=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Start=C2=A0 =C2=A0 =C2=A0 En= d Sectors=C2=A0 Size Type
/dev/nvme1n1p1=C2=A0 =C2=A0 =C2=A0 =C2=A03=C2=A0 =C2=A0 =C2=A0 129=C2=A0 = =C2=A0 =C2=A0127 63.5K FreeBSD boot
/dev/nvme1n1p2=C2=A0 =C2=A0 =C2=A0130=C2=A0 =C2=A0 66713=C2=A0 =C2=A066584 = 32.5M EFI System
/dev/nvme1n1p3=C2=A0 =C2=A066714=C2=A0 2163865 2097152=C2=A0 =C2=A0 1G Free= BSD swap
/dev/nvme1n1p4 2163866 10552473 8388608=C2=A0 =C2=A0 4G FreeBSD UFS

The bhyve invocation is:
bhyve -c 2 -m 2G -Hw \
=C2=A0 =C2=A0 -s 0,hostbridge \
=C2=A0 =C2=A0 -s 4:0,virtio-blk,/dev/zvol/zroot/vms/test-vm/disk0 \
=C2=A0 =C2=A0 -s 5:0,nvme,/dev/zvol/zroot/vmvol/disk0 \
=C2=A0 =C2=A0 -s 6:0,nvme,/dev/zvol/zroot/vmvol/gptdisk \
=C2=A0 =C2=A0 -s 7:0,virtio-net,tap0,mac=3D58:9c:fc:0b:ed:d6
=C2=A0 =C2=A0 -s 31,lpc \
=C2=A0 =C2=A0 -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
=C2=A0 =C2=A0 -l com1,/dev/nmdm-test-vm.1A \
=C2=A0 =C2=A0 test-vm

Note that my experiment is different from what you are doing. But I
would expect a Zvol and raw block device to behave the same as a
backing-store for an emulated NVMe drive in bhyve. My experiment ran
on -current, but the behavior in this area should be identical to the
13.0-p8 version you are using.

Does the output on the host of
=C2=A0# hd -n 256 /dev/nvd0
match the output on the guest of
=C2=A0# hd -n 256 /dev/nvme0n1
?


--
Mario.
--00000000000052d23605daab0835--