From nobody Fri Feb 03 13:07:42 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4P7bYx3PFXz2p5Mp for ; Fri, 3 Feb 2023 13:08:21 +0000 (UTC) (envelope-from marietto2008@gmail.com) Received: from mail-ej1-x636.google.com (mail-ej1-x636.google.com [IPv6:2a00:1450:4864:20::636]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4P7bYw3nBzz3xmQ for ; Fri, 3 Feb 2023 13:08:20 +0000 (UTC) (envelope-from marietto2008@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=OnGZA8yM; spf=pass (mx1.freebsd.org: domain of marietto2008@gmail.com designates 2a00:1450:4864:20::636 as permitted sender) smtp.mailfrom=marietto2008@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-ej1-x636.google.com with SMTP id me3so15162209ejb.7 for ; Fri, 03 Feb 2023 05:08:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=4+MWOLqU4uNgcthREDDvVeAbpux5YkqI0KKP+ZlL1ls=; b=OnGZA8yMVZgAGjJRiRGNtOspWhm4UL5lcfDmihtlLFikBP05/B2PAXJIfP7MwV5b6t +IDgdH/C+9FrXH/psQ/w61S9/MKDYoT7ko1fT8Gv+V5jU3xRVGm8Pr48ZoFE+LAgvdUh WfxQbz6LaFpXtGUE80vr7lMgOv+q9FwiaM4piRP2HkVcSAGIwZy6g597D4Qn+JNDCArY PuzNzw0Flt1RWMRoz2dePra4RrvMLrCVaCXEB78GWul8je8623JIH1dYcWPAGlKkkzH8 lnbV8MvjfvzgZ1Bvy+ThHC7JbQIsknTzWw/SQ9uiZfvY0oN9Bd4N9anhDBHSvEAOojJp EGow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=4+MWOLqU4uNgcthREDDvVeAbpux5YkqI0KKP+ZlL1ls=; b=MQ0Hgr8kB5Ytwv/O/tEjEzlPOpFncVZJZL4enPbdZDKigJsv8YdMpNH/lNHknMZ4+L Nh98Yz2N7hvWRoSV4aImMQHPqz6EkMcoeLE41R6jGyBW8qu+3TSJMapPvdsKuWXsUskE RT8S0oJahJB6ioOssqx3YxoBflARSZHc2bwVOxIhaIty5EsDpsD6cqr1DIl1rv/ZaXM2 W5MiISpUJYcPkze1kUny7RQ9GvmzMdtyaZvF13xgObdHIvlyJLrTqhdj6gJudaGjwemc y0Hd7Rh2DTsa6dw8xMO0Q9aSFyoegP7wM2mS3pb7RZm2l8ZmQ1MyfCAFCzZx72L5VNdE u83w== X-Gm-Message-State: AO0yUKXhgebDe+4M26X/pcga7nSN+NIRs8oNWZNba1qQpmZJtXNZ/nKE 2+i5/N1OkXZIJcdHg5z6XqQfVhYMq8f/fFLVFhoj15k9LdY51Q== X-Google-Smtp-Source: AK7set9rcdKHWqNcpxBZy6CU026pRZ/cgC+WVM6303XLp30HwGuDPN05pAr9EsLlPdoEe3T9MT+hDSPAHhM7CsZKuJI= X-Received: by 2002:a17:907:cb25:b0:880:b580:d93 with SMTP id um37-20020a170907cb2500b00880b5800d93mr3044801ejc.276.1675429699107; Fri, 03 Feb 2023 05:08:19 -0800 (PST) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 References: <09A3D05B-E269-437F-8ACB-A150EBCACF4E@dons.net.au> <128F6D81-D71B-4BC7-9143-8205E826088F@dons.net.au> In-Reply-To: <128F6D81-D71B-4BC7-9143-8205E826088F@dons.net.au> From: Mario Marietto Date: Fri, 3 Feb 2023 14:07:42 +0100 Message-ID: Subject: Re: devctl: Failed to detach pci0:1:0:0: Device busy / devctl: Failed to set pci0:1:0:0 driver to ppt: Device busy To: "Daniel O'Connor" Cc: freebsd-hackers Content-Type: multipart/alternative; boundary="0000000000000fc01705f3cb609f" X-Spamd-Result: default: False [-3.94 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.94)[-0.940]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; MID_RHS_MATCH_FROMTLD(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::636:from]; DKIM_TRACE(0.00)[gmail.com:+]; RCVD_COUNT_TWO(0.00)[2]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_FROM(0.00)[gmail.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; TO_DN_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-Rspamd-Queue-Id: 4P7bYw3nBzz3xmQ X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N --0000000000000fc01705f3cb609f Content-Type: text/plain; charset="UTF-8" Hello Daniel and everyone. ---> If you want to pass through the video card (as evidenced by devctl and pptdevs) then why have the nvidia driver installed? Why did you say this ? Are you aware of some bug that prevents a pci device from being passed if its driver is installed ? Maybe there is. Infact I made some progress. I have commented this line on /etc/rc.conf : #kld_list="nvidia nvidia-modeset" and I've been able to detach and reattach the 4 slots of the 2080 ti. I think that if I load the nvidia modules on the rc.conf,they interfere with a later attaching / detaching of 1 or 2 slots of the gpu (hdac and vgapci). Do you think that it could be caused by a bug on the nvidia driver or in the bhyve source code ? What do you suggest I do ? Il giorno mer 1 feb 2023 alle ore 13:49 Daniel O'Connor ha scritto: > > > > On 1 Feb 2023, at 21:41, Mario Marietto wrote: > > ---> Do you have 'vmm_load="YES"' in loader.conf ? > > > > yes. > > > > ---> If you want to pass through the video card (as evidenced by devctl > and pptdevs) then why have the nvidia driver installed? > > > > because I have 3 gpus on my PC. I use the intel gpu on the host ; I > would like to use the nvidia gtx 1060 within a bhyve Linux vm and the 2080 > ti with the Linuxulator to run stable diffusion,or vice versa. Stable > diffusion needs pytorch + cuda and they need the nVidia driver installed on > FreeBSD and the modules loaded. > > > > My goal is to run stable diffusion and a bhyve / linux vm +my nvidia > gpus attached. > > I note that https://wiki.freebsd.org/bhyve/pci_passthru says "Note: VGA / > GPU pass-through devices are not currently supported" so perhaps that is > why it is ignoring your pptdevs line. > > Although that said some googling suggests that is not the case so I don't > know, unfortunately I haven't tried it myself. > > There are some threads about it on the forums, eg: > https://forums.freebsd.org/threads/bhyve-gpu-pass-through.83152/ > > Those suggest that modifications to bhyve are required before it will work > properly but none of them are for vmm that I can see. > > -- > Daniel O'Connor > "The nice thing about standards is that there > are so many of them to choose from." > -- Andrew Tanenbaum > > -- Mario. --0000000000000fc01705f3cb609f Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hello Daniel and everyone.

 ---> If you want to pass through the video card (as evidenced by devctl = and pptdevs) then why have the nvidia driver installed?

Why did you say this ? Are you aware of some bug that prevents a pci = device from being passed if its driver is installed ?

Maybe there is. Infact I made some progress. I have commented this= line on /etc/rc.conf :

#kld_list=3D"nvidia nvidia-modeset"

and I've been able to detach and reattach the 4 slots of the 2080 ti. I=20 think that if I load the nvidia modules on the rc.conf,they interfere=20 with a later attaching / detaching of 1 or 2 slots of the gpu (hdac and=20 vgapci). Do you think that it could be caused by a bug on the nvidia driver= or in the bhyve source code ? What do you suggest I do ?
=

Il giorno mer 1 feb 2023 alle ore 13:49 Daniel O'Connor <darius@dons.net.au> ha scritto:
=


> On 1 Feb 2023, at 21:41, Mario Marietto <marietto2008@gmail.com> wrote:
> ---> Do you have 'vmm_load=3D"YES"' in loader.con= f ?
>
> yes.
>
> ---> If you want to pass through the video card (as evidenced by de= vctl and pptdevs) then why have the nvidia driver installed?
>
> because I have 3 gpus on my PC. I use the intel gpu on the host ; I wo= uld like to use the nvidia gtx 1060 within a bhyve Linux vm and the 2080 ti= with the Linuxulator to run stable diffusion,or vice versa. Stable diffusi= on needs pytorch + cuda and they need the nVidia driver installed on FreeBS= D and the modules loaded.
>
> My goal is to run stable diffusion and a bhyve / linux vm +my nvidia g= pus attached.

I note that https://wiki.freebsd.org/bhyve/pci_passthru says "Note: VGA / GPU pass-through devices are not currently suppor= ted" so perhaps that is why it is ignoring your pptdevs line.

Although that said some googling suggests that is not the case so I don'= ;t know, unfortunately I haven't tried it myself.

There are some threads about it on the forums, eg:
https://forums.freebsd.org/threads/b= hyve-gpu-pass-through.83152/

Those suggest that modifications to bhyve are required before it will work = properly but none of them are for vmm that I can see.

--
Daniel O'Connor
"The nice thing about standards is that there
are so many of them to choose from."
-- Andrew Tanenbaum



--
Mario.
--0000000000000fc01705f3cb609f--