From nobody Tue Sep 27 12:21:18 2022 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4McJdw5Cmmz4crq8 for ; Tue, 27 Sep 2022 12:21:56 +0000 (UTC) (envelope-from marietto2008@gmail.com) Received: from mail-ed1-x52e.google.com (mail-ed1-x52e.google.com [IPv6:2a00:1450:4864:20::52e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4McJdw2Fzwz3Lsq for ; Tue, 27 Sep 2022 12:21:56 +0000 (UTC) (envelope-from marietto2008@gmail.com) Received: by mail-ed1-x52e.google.com with SMTP id u24so12941170edb.9 for ; Tue, 27 Sep 2022 05:21:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=to:subject:message-id:date:from:mime-version:from:to:cc:subject :date; bh=Uf/Gp1NQuOYKxCaAj1JK1Hm8YbNnAVLmaB+e93ubdTo=; b=KN1tDRp2ZeZSP/AqSsQ1M79VgACOEeSBLrcyYjvgmTx2Qvc0/xp6mOLMTqfQMKljEP DyhiqIXJS6Q8mXlZ/v93p1AgCMQFZxQWOU7gWFm96pI1zmozc1/bUfRVNme1hTL9qnVJ 1aB4HERLSOI7VF8NY9h0EHzmW2RrR3lNMceEBrXIWy/f2joKZJlDOR3czcmJD8NEL+Gg nBr4nslFkMULSiSn5dZ3UvzxOg7yobY/oUM8E7M0KkMNKOQ/SV1nVxIPGyMP8NIIzHpI op33Na0Jua2GfZMLqYgv1SUUuHP9MDoenLdvGo1Wt2+SiGa9EQO5JSlknIH9pmnNC2pT rMIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date; bh=Uf/Gp1NQuOYKxCaAj1JK1Hm8YbNnAVLmaB+e93ubdTo=; b=Tl1GDBouO6FwAkE+gUs4ths7LuZwcZyv6GN4uZmiLDIERWpK4ly6HUUwZmMjMCw/zs k70VjS4ITk5sE0lJoR2/Dewt3heu7azcf1N4RRiVSoqJGFsSqXnnB4t0nYH3tw5p5Kmj tE1pwzx6fw9iG7jiZOdH6NOmT2DusCSLataaXlwvu42RccmQ4t1FI2PkxbeKVHaeta+d tc7jKQRXgj/74CUoxTSsT8KuKlA3rRBgKmIxcpNxTJ6ehcFDHSkH0kXf5870mJrZK8K3 dixv9PwfXgZ9ZA+BZPcpIOy3y2J8iFF/c3inKoeWkwlOGFL19AJCy1wERzdhkDi0VxP6 HnIA== X-Gm-Message-State: ACrzQf1oWeTw4Gx2XpWyqX+8UP0PZwPnHPYVMiyl7qB3mSLY6UaLNKZN LYGWbNSq1CNMOzw++pa8p7+a8KmBaoW2lDe9TyM1qae1Ymw= X-Google-Smtp-Source: AMsMyM6w+xT02Nafki7OA8ftOh/BB6wmu6hvoL457n2S1V0KLmZe5nuyj5cNCm3VQj4mXJ7ubIuwL45HeyEfPSDV9gU= X-Received: by 2002:aa7:c74c:0:b0:44e:a7b9:d5c9 with SMTP id c12-20020aa7c74c000000b0044ea7b9d5c9mr27922532eds.19.1664281314540; Tue, 27 Sep 2022 05:21:54 -0700 (PDT) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 From: Mario Marietto Date: Tue, 27 Sep 2022 14:21:18 +0200 Message-ID: Subject: My nvidia gpu can't be passed through from Linux (qemu+kvm) to the 2. guest OS (Puppy Linux) that I tried to virtualize on the 1. (FreeBSD 13.1) To: FreeBSD virtualization Content-Type: multipart/alternative; boundary="0000000000008f3feb05e9a7b03d" X-Rspamd-Queue-Id: 4McJdw2Fzwz3Lsq X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=KN1tDRp2; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of marietto2008@gmail.com designates 2a00:1450:4864:20::52e as permitted sender) smtp.mailfrom=marietto2008@gmail.com X-Spamd-Result: default: False [-3.00 / 15.00]; HTTP_TO_IP(1.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-0.996]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::52e:from]; RCPT_COUNT_ONE(0.00)[1]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; FREEMAIL_FROM(0.00)[gmail.com]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; DKIM_TRACE(0.00)[gmail.com:+]; TO_DN_ALL(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; ARC_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCVD_COUNT_TWO(0.00)[2] X-ThisMailContainsUnwantedMimeParts: N --0000000000008f3feb05e9a7b03d Content-Type: text/plain; charset="UTF-8" Hello. I've configured Xubuntu 22.04 so that it can allow me to pass thru my nvidia gpu on the guest os vms. This time I've chosen freebsd 13.1 as a guest,because I was curious to see if bhyve supports the passthrough of my nvidia gpu within another guest os (Puppy Linux) that I have virtualized with bhyve. This is how I have configured the FreeBSD 13.1 guest vm : /boot/loader.conf vmm_load="YES" nmdm_load="YES" if_tap_load="YES" if_bridge_load="YES" bridgestp_load="YES" kern.geom.label.disk_ident.enable="0" kern.geom.label.gptid.enable="0" kern.racct.enable=1 aio_load="YES" cryptodev_load="YES" zfs_load="YES" verbose_loading="YES" pptdevs="8/0/0 9/0/0 10/0/0 11/0/0" pciconf -vl says that pptdevs reserved correctly the addresses 8/0/0 9/0/0 10/0/0 11/0/0 which belong to the nvidia GPU : ppt0@pci0:8:0:0: class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de device=0x1e04 subvendor=0x19da subdevice=0x2503 vendor = 'NVIDIA Corporation' device = 'TU102 [GeForce RTX 2080 Ti]' class = display subclass = VGA ppt1@pci0:9:0:0: class=0x040300 rev=0xa1 hdr=0x00 vendor=0x10de device=0x10f7 subvendor=0x19da subdevice=0x2503 vendor = 'NVIDIA Corporation' device = 'TU102 High Definition Audio Controller' class = multimedia subclass = HDA ppt2@pci0:10:0:0: class=0x0c0330 rev=0xa1 hdr=0x00 vendor=0x10de device=0x1ad6 subvendor=0x19da subdevice=0x2503 vendor = 'NVIDIA Corporation' device = 'TU102 USB 3.1 Host Controller' class = serial bus subclass = USB ppt3@pci0:11:0:0: class=0x0c8000 rev=0xa1 hdr=0x00 vendor=0x10de device=0x1ad7 subvendor=0x19da subdevice=0x2503 vendor = 'NVIDIA Corporation' device = 'TU102 USB Type-C UCSI Controller' At this point,inside FreeBSD 13.1 guest OS,I tried to virtualize another OS,the puppy Linux : bhyve -S -c sockets=1,cores=1,threads=1 -m 2G -w -H -A \ -s 0,hostbridge \ -s 1,ahci-cd,/home/marietto/Desktop/bhyve/Files/fossapup64-9.5.iso,bootindex=1 \ -s 2,virtio-blk,/home/marietto/Desktop/bhyve/Files/puppy.img,bootindex=2 \ -s 8:0,passthru,8/0/0 \ -s 8:1,passthru,9/0/0 \ -s 8:2,passthru,10/0/0 \ -s 8:3,passthru,11/0/0 \ -s 10,virtio-net,tap18 \ -s 11,virtio-9p,sharename=/ \ -s 29,fbuf,tcp=0.0.0.0:5918,w=800,h=600,wait \ -s 30,xhci,tablet \ -s 31,lpc \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \ vm0:18 < /dev/null & sleep 2 && vncviewer 0:18 unfortunately it gives this error : *bhyve: PCI device at 8/0/0 is not using the ppt(4) driverdevice emulation initialization error: Device busy* I tried to remove the slots related to the passthru of the gpu and puppy booted like a charm,like this one : bhyve -S -c sockets=1,cores=1,threads=1 -m 2G -w -H -A \ -s 0,hostbridge \ -s 1,ahci-cd,/home/marietto/Desktop/bhyve/Files/fossapup64-9.5.iso,bootindex=1 \ -s 2,virtio-blk,/home/marietto/Desktop/bhyve/Files/puppy.img,bootindex=2 \ -s 10,virtio-net,tap18 \ -s 11,virtio-9p,sharename=/ \ -s 29,fbuf,tcp=0.0.0.0:5918,w=800,h=600,wait \ -s 30,xhci,tablet \ -s 31,lpc \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \ vm0:18 < /dev/null & sleep 2 && vncviewer 0:18 So,where could be the error in this specific scenario ? Nested vm works,but I can't pass through the gpu,even if the host OS (xubuntu) makes it available to the guest. -- Mario. --0000000000008f3feb05e9a7b03d Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

Hello.

I've configured Xubuntu 22.04 so that it can allow me to pass thru my =20 nvidia gpu on the guest os vms. This time I've chosen freebsd 13.1 as a= =20 guest,because I was curious to see if bhyve supports the passthrough of my nvidia gpu within another guest os (Puppy Linux) that I have =20 virtualized with bhyve. This is how I have configured the FreeBSD 13.1=20 guest vm :


/boot/loader.conf


vmm_load=3D"YES"
nmdm_load=3D"YES"
if_tap_load=3D"YES"
if_bridge_load=3D"YES"
bridgestp_load=3D"YES"
kern.geom.label.disk_ident.enable=3D"0"
kern.geom.label.gptid.enable=3D"0"
kern.racct.enable=3D1
aio_load=3D"YES"
cryptodev_load=3D"YES"
zfs_load=3D"YES"
verbose_loading=3D"YES"
pptdevs=3D"8/0/0 9/0/0 10/0/0 11/0/0"


pciconf -vl says that pptdevs reserved corr= ectly the addresses 8/0/0 9/0/0 10/0/0 11/0/0 which belong to the nvidia GP= U :


p=
pt0@pci0:8:0:0:    class=3D0x030000 rev=3D0xa1 hdr=3D0x00 vendor=3D0x10de d=
evice=3D0x1e04 subvendor=3D0x19da subdevice=3D0x2503
    vendor     =3D 'NVIDIA Corporation'
    device     =3D 'TU102 [GeForce RTX 2080 Ti]'
    class      =3D display
    subclass   =3D VGA

ppt1@pci0:9:0:0:    class=3D0x040300 rev=3D0xa1 hdr=3D0x00 vendor=3D0x10de =
device=3D0x10f7 subvendor=3D0x19da subdevice=3D0x2503
    vendor     =3D 'NVIDIA Corporation'
    device     =3D 'TU102 High Definition Audio Controller'
    class      =3D multimedia
    subclass   =3D HDA

ppt2@pci0:10:0:0:    class=3D0x0c0330 rev=3D0xa1 hdr=3D0x00 vendor=3D0x10de=
 device=3D0x1ad6 subvendor=3D0x19da subdevice=3D0x2503
    vendor     =3D 'NVIDIA Corporation'
    device     =3D 'TU102 USB 3.1 Host Controller'
    class      =3D serial bus
    subclass   =3D USB

ppt3@pci0:11:0:0:    class=3D0x0c8000 rev=3D0xa1 hdr=3D0x00 vendor=3D0x10de=
 device=3D0x1ad7 subvendor=3D0x19da subdevice=3D0x2503
    vendor     =3D 'NVIDIA Corporation'
    device     =3D 'TU102 USB Type-C UCSI Controller'


At this point,inside FreeBSD 13.1 guest OS,= I tried to virtualize another OS,the puppy Linux :


=
bhyve -S -c sockets=3D1,cores=
=3D1,threads=3D1 -m 2G -w -H -A \
-s 0,hostbridge \
-s 1,ahci-cd,/home/marietto/Desktop/bhyve/Files/fossapup64-9.5.iso,bootinde=
x=3D1 \
-s 2,virtio-blk,/home/marietto/Desktop/bhyve/Files/puppy.img,bootindex=3D2 =
\
-s 8:0,passthru,8/0/0 \
-s 8:1,passthru,9/0/0 \
-s 8:2,passthru,10/0/0 \
-s 8:3,passthru,11/0/0 \
-s 10,virtio-net,tap18 \
-s 11,virtio-9p,sharename=3D/ \
-s 29,fbuf,tcp=3D0.0.0.0:5918,w=3D800,h=
=3D600,wait \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \
vm0:18 < /dev/null & sleep 2 && vncviewer 0:18


unfortunately it gives this error :

bhyve: PCI device at 8/0/0 is not using the ppt(4) driverdevice= emulation initialization error: Device busy

I tried to remove= the slots related to the passthru of the gpu and puppy booted like a charm= ,like this one :


bhyve -S -c sockets=3D1,cores=3D1,threads=3D1 -m 2G -w -H -A \ -s 0,hostbridge \ -s 1,ahci-cd,/home/marietto/Desktop/bhyve/Files/fossapup64-9.5.iso,bootinde= x=3D1 \ -s 2,virtio-blk,/home/marietto/Desktop/bhyve/Files/puppy.img,bootindex=3D2 = \ -s 10,virtio-net,tap18 \ -s 11,virtio-9p,sharename=3D/ \ -s 29,fbuf,tcp=3D0.0.0.0:5918,w=3D800,h= =3D600,wait \ -s 30,xhci,tablet \ -s 31,lpc \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \ vm0:18 < /dev/null & sleep 2 && vncviewer 0:18


So,where could be the error in this specific scenario ? Nested vm works,but I=20 can't pass through the gpu,even if the host OS (xubuntu) makes it=20 available to the guest.

--
Mario.
--0000000000008f3feb05e9a7b03d--