Re: devctl: Failed to detach pci0:2:0:0: Device busy
- Go to: [ bottom of page ] [ top of archives ] [ this month ]
Date: Fri, 03 Feb 2023 17:09:40 UTC
Hello, bhyve and / or FreeBSD developers and lovers. I’m trying to pass my GTX 1060 from my host os (FreeBSD 13.1-RELEASE) to a Linux/Ubuntu VM virtualized with bhyve,but unsuccessfully. Last year I tried to do the same with another GPU that I have on my PC,the RTX 2080 ti and with the help of @generix <https://forums.developer.nvidia.com/u/generix>,after a tricky trouble shooting we have been able to do that. Point is that the same patches which worked (and that works right now) for the 2080 ti aren’t working for the GTX 1060. The error that I see is the same that I saw when I have cooperated with Generix for passing through the 2080 ti. I really would like to know which differences there are between the two gpus because for the 1060 I'm using the same new driver that I’m using for the 2080 ti and the latter one can be passed without problems. Anyway,I would know what I should do to fix that error. For this matter I've created a new post on the nvidia forum. If someone would like to give it a look,it is here : https://forums.developer.nvidia.com/t/rminitadapter-failed-error-produced-when-i-try-to-passthru-my-gtx-1060-to-a-linux-ubuntu-bhyve-vm-instead-my-rtx-2080ti-is-passed-correctly/241594 Il giorno ven 3 feb 2023 alle ore 15:21 Mario Marietto < marietto2008@gmail.com> ha scritto: > Very interesting behavior. I've added this new line in /etc/rc.conf : > (until some minutes ago I didn't use it because I was using the intel > driver,installed by the package : xf86-video-intel-2.99.917.916_2,1) > > kld_list="i915kms acpi_video" > > and this line in /boot/loader.conf : > > pptdevs="1/0/0 1/0/1 2/0/0 2/0/1 2/0/2 2/0/3 5/0/0" > > and boom. pptdevs worked again: > > # pciconf -l > > ppt0@pci0:1:0:0: class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de > device=0x1c02 subvendor=0x19da subdevice=0x2438 > ppt1@pci0:1:0:1: class=0x040300 rev=0xa1 hdr=0x00 vendor=0x10de > device=0x10f1 subvendor=0x19da subdevice=0x2438 > ppt2@pci0:2:0:0: class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de > device=0x1e04 subvendor=0x19da subdevice=0x2503 > ppt3@pci0:2:0:1: class=0x040300 rev=0xa1 hdr=0x00 vendor=0x10de > device=0x10f7 subvendor=0x19da subdevice=0x2503 > ppt4@pci0:2:0:2: class=0x0c0330 rev=0xa1 hdr=0x00 vendor=0x10de > device=0x1ad6 subvendor=0x19da subdevice=0x2503 > ppt5@pci0:2:0:3: class=0x0c8000 rev=0xa1 hdr=0x00 vendor=0x10de > device=0x1ad7 subvendor=0x19da subdevice=0x2503 > ppt6@pci0:5:0:0: class=0x0c0330 rev=0x10 hdr=0x00 vendor=0x1b73 > device=0x1100 subvendor=0x1b73 subdevice=0x1100 > > So,maybe there is some incompatibility between the intel and the pptdevs > driver ? take also in consideration that on xorg.cong I'm still using the > intel driver : > > Section "Device" > Identifier "Card0" > Driver "intel" > BusID "PCI:0:2:0" > EndSection. > > It's a confusing situation. > > Il giorno ven 3 feb 2023 alle ore 14:13 Mario Marietto < > marietto2008@gmail.com> ha scritto: > >> I've sent an email to the hackers ML with this content : >> >> > On 30 Jan 2023, at 05:37, Mario Marietto <marietto2008@gmail.com> >> wrote: >> > In some FreeBSD 13.1 machines I have the problem below,in some others I >> don't have it. I would like to know what the causes could be and how to fix >> it. >> > >> > root@marietto:/usr/home/marietto # devctl detach pci0:1:0:0 >> > >> > devctl: Failed to detach pci0:1:0:0: Device busy >> > >> > root@marietto:/usr/home/marietto # devctl detach pci0:2:0:0 >> > >> > devctl: Failed to detach pci0:2:0:0: Device busy >> > >> > Not even it works if instead of detach them, I try to attach them >> directly to the ppt driver : >> > >> > root@marietto:/usr/home/marietto/bhyve/Files # devctl set driver >> pci0:2:0:0 ppt >> > devctl: Failed to set pci0:2:0:0 driver to ppt: Device busy >> > >> > root@marietto:/usr/home/marietto/bhyve/Files # devctl detach >> pci0:2:0:0 >> > devctl: Failed to detach pci0:2:0:0: Device busy >> > >> > Pci addresses 1:0:0 and 2:0:0 belong to the two GPUs that I have on my >> PC : >> > >> > vgapci0@pci0:1:0:0: class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de >> device=0x1c02 subvendor=0x19da subdevice=0x2438 >> > vgapci1@pci0:2:0:0: class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de >> device=0x1e04 subvendor=0x19da subdevice=0x2503 >> > >> > Actually I have commented this line on /boot/loader.conf,because it >> makes no difference if I keep it uncommented or not. It is totally ignored. >> > >> > #pptdevs="1/0/0 1/0/1 2/0/0 2/0/1 2/0/2 2/0/3" >> >> and George replied to me like this : >> >> Do you have 'vmm_load="YES"' in loader.conf? >> >> If not I am not sure it will have an effect - vmm must be loaded early >> otherwise the card will be grabbed by the vgapci driver rather than be put >> under ppt control. >> >> I think this is an interesting answer. Yes,I keep vmm_load="YES" in >> loader.conf,BUT putting the 4 slots under ppt control using the loader.conf >> did not work for me for months. We haven't taken this into consideration. >> And you didn't reply when I've asked you if you have modified the bhyve >> source code,because putting the pci addresses under ppt using the >> loader.conf always worked for me. I would like to understand why it does >> not work anymore. This is important to understand why I'm not able to >> attach / detach the nVidia driver from the host to the guest os. And maybe >> George's consideration is relevant. >> >> Il giorno ven 3 feb 2023 alle ore 12:11 Corvin Köhne < >> C.Koehne@beckhoff.com> ha scritto: >> >>> >>> - ---> Figure out if you can blacklist your gpu in the nvidia >>> driver. So, the nVidia driver never gets attached to your device. >>> >>> >>> - nope. I need to attach the nVidia driver to my gpu if I want to >>> run stable diffusion + pytorch + cuda on the Linuxulator on FreeBSD. I'm >>> thinking of installing Stable Diffusion within the Linux VM,instead of >>> doing it on FreeBSD. >>> >>> >>> - ---> Wait for drm drivers and use the nouveau driver. Nouveau >>> isn’t maintained by nVidia, so it’s more likely to get fixed. >>> >>> >>> - The nouveau driver is always bugged and it never worked well. I >>> wonder if it is still supported. It has even limited features. >>> >>> >>> >>> Afaik, you don’t have other options, sry. I’m no gpu driver developer, >>> so I can’t help you. >>> >>> Btw: Do you refer to linux nouveau or freebsd nouveau? Freebsd made much >>> progress in porting linux drm to freebsd with 14.0. So, if latest linux >>> nouveau fits your requirements, it’s likely that freebsd 14.0 may fit them >>> too. >>> >>> >>> >>> - Are the following commands enough to update the bhyve source code >>> on my machine ? take in consideration that I need to grab only your new >>> last patch. >>> >>> >>> - cd /usr/src >>> >>> >>> - git checkout -f origin/phab/corvink/13.1/nvidia-wip >>> >>> >>> - # cd /wherever/you/put/my/Makefile >>> >>> # make -j 8 >>> >>> >>> >>> My latest patch is a mouse driver patch for the host. So, no, you have >>> to use: >>> >>> cd /usr/src >>> >>> git fetch >>> >>> git checkout -f origin/phab/corvink/13.1/nvidia-wip >>> >>> make kernel >>> >>> reboot >>> >>> >>> >>> >>> >>> Best regards >>> >>> Corvin >>> >>> >>> >>> >>> >>> This email contains confidential information. If you have received it in >>> error, you must not read, use, copy or pass on this e-mail or its >>> attachments. >>> If you have received the e-mail in error, please inform me immediately >>> by reply e-mail and then delete this e-mail from your system. Thank you >>> >>> Diese E-Mail enthaelt vertrauliche Informationen. Sollten Sie sie >>> irrtuemlich erhalten haben, duerfen Sie diese E-Mail oder ihre Anhaenge >>> nicht lesen, verwenden, kopieren oder weitergeben. >>> Sollten Sie die Mail versehentlich erhalten haben, teilen Sie mir dies >>> bitte umgehend per Antwort-E-Mail mit und loeschen Sie diese E-Mail dann >>> aus Ihrem System. Vielen Dank >>> >>> Beckhoff Automation GmbH & Co. KG | Managing Director: Dipl. Phys. Hans >>> Beckhoff >>> Registered office: Verl, Germany | Register court: Guetersloh HRA 7075 >>> >>> >> >> -- >> Mario. >> > > > -- > Mario. > -- Mario.