Re: GPU Passthrough on FreeBSD 14.3 (AMD Radeon RX 6900 XT and Windows 10 Pro)

Hey all, I hope everyone is doing well. I wanted to give some final updates since I feel I'm basically done with my research, set up, and testing. I'm pleased to say that I'm officially calling my setup as "stable". For the past week, I've been testing the VM by either playing some games on it for a few hours, or leaving the VM on idle, and so far I haven't really experienced any VM crashes. The games seem to be working smoothly when I'm playing them, and I don't get any sudden VM crashes. I left the VM on idle for the past 1d20h and no issues. I wanted to finish working on my rc.d start up/shutdown script so I had to bring the VM down to do that :P. There were some instances where the VM wouldn't really display any output and would effectively be frozen when I rebooted the machine multiple consecutive times during my testing, so much so that doing a "service vm_gaming restart" (or stop) would just leave the script at the "Waiting for PIDs" message. In this case I had to do a "bhyvectl --vm=gaming --force-poweroff" to kill it. The subsequent start of the VM then seemed to work correctly and video was working again. This doesn't happen all the time but it could happen so I wanted to mention that for the record. Once the VM is up, it's pretty stable. I do see the windows start up screen always say "Getting devices ready" and then it proceeds to complete the boot with no issues. Not sure what that's about but there are no negative consequences.

In regards to my research regarding the graceful shutdown, given that throughout all of many days of testing, my VM hasn't gotten any disk corruption, I believe that the KEYWORD: shutdown (or KEYWORD: nojail shutdown) have been working correctly. When doing my testing, I do see that if I do a "service vm_gaming stop" explicitly, I see Windows 10 showing the light blue screen with the "Shutting down" message. When I do "shutdown -r now" I sometimes see the same light blue screen, but usually I don't see the screen, what I do see though is that if I move my mouse cursor to the middle of the screen, once I trigger the shutdown, I would see the light blue circle/loop around the mouse cursor start to spin. This happens every single time I run the "shutdown -r now" command. So this means that the shutdown signal is properly being sent, and my theory is that the GPU/OS is just not reflecting that screen update due to the speed at which things are happening. I also do see that the time it takes to reboot after the shutdown -r now is called takes a little bit of time, which I believe is also partly the time it takes for the system to wait for the bhyve vm to shutdown and for resource to possibly be cleaned (bhyvectl ... destroy). So I believe I'm done with the main research, experimentation, and testing. I've updated my blog post as well, and below you can find my updated rc.d script.  I saw Corvin recently pushed some updates to CURRENT to get NVIDIA passthrough working (PIN related work). I do have my Razer Blade 15" (2020) that has an Intel proc and NVIDIA GTX 2060. In the future I may try and install FreeBSD on there to test the Intel/NVIDIA side of things, but that's not something I'm planning on doing any time soon.

Take care all and I hope all of this helps y'all have some fun and relax. Stay safe, Jonathan!


---- updated script

#!/bin/sh

# PROVIDE: vm_gaming
# REQUIRE: LOGIN FILESYSTEMS
# KEYWORD: shutdown

. /etc/rc.subr

name="vm_gaming"
rcvar="vm_gaming_enable"
start_cmd="${name}_start"
stop_cmd="${name}_stop"
stop_postcmd="${name}_poststop"
restart_cmd="${name}_restart"
status_cmd="${name}_status"
pidfile="/var/run/${name}.pid"

vm_path="/atlantis/vms/gaming"
vm_name="gaming"

load_rc_config $name

: ${vm_gaming_enable:="NO"}

vm_gaming_start()
{
	daemon \
		-o /var/log/${name}.log \
		-p "${pidfile}" \
		bhyve -AHPSw -c sockets=1,cores=16,threads=1 -m 32G \
		-s 0,hostbridge \
		-s 1,nvme,${vm_path}/disk0.img \
		-s 3:0,passthru,3/0/0 \
		-s 3:1,passthru,3/0/1 \
		-s 13:0,passthru,13/0/0 \
		-s 30,xhci,tablet \
		-s 31,lpc \
		-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,fwcfg=qemu \
		-o console=stdio \
		${vm_name}
}

vm_gaming_stop()
{
	if [ ! -f "${pidfile}" ]; then
		echo "the gaming vm isn't running!"
		return
	fi

	# Send SIGTERM twice to make sure Windows listens to
	# the ACPI shutdown signal.
	pid="$(cat ${pidfile})"
	kill "${pid}"
	kill "${pid}"

	# Wait a bit for the guest to shutdown properly before
	# we continue shutting down the host.
	wait_for_pids "${pid}"
}

vm_gaming_poststop()
{
	bhyvectl --vm="${vm_name}" --destroy
}

vm_gaming_restart()
{
	# NOTE: AMD users will most likely experience the famous
	# AMD Hardware Reset Bug. This means that after you reboot
	# the guest, you most likely won't have video out. If this
	# happens, you'll need to restart the host. Sometimes this
	# command will work for AMD users though. So you can try a
	# restart and see if it works.
	vm_gaming_stop
	vm_gaming_start
}

vm_gaming_status()
{
	if [ -f "${pidfile}" ]; then
		echo "$(cat ${pidfile})"
	fi
}

run_rc_command "$1"