Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2)

[soralx at cydem.org]

> As far as I can tell it's the Hypervisor extension flags list. The lack 
> of these extensions/optimisations might explain why your FreeBSD VM
> runs slow 

The guest isn't slow, actually -- just the `nvidia-smi` tool was
much slower than normal to produce output. CPU speed in the guest
is less than 2% slower than bare hardware. Memory bandwidth is
over 2.5 times slower. [0]  Still, compiling world and ports feels
quite snappy. Disk I/O is not bad, too! (but latencies are bit
high) [1]

> However, even with reapplying the changes to vmm.ko to hide/remove the 
> 0x40000000 CPUID support and CPUID2_HV, I still have the same 
> "RmInitAdapter failed" issue.

Well, I, too, couldn't get nVidia driver to work at all in Linux.
The driver loaded, but when trying to use it, gave the "RmInitAdapter
failed! (0x53:0x3:1856)" errors. Is is the same in your case? Or
the driver does not load at all (like I experienced with Quadro 600)?
Maybe I should try the same driver version in Linux as in FreeBSD...

> Allegedly[0] nVidia VM checking came in with driver version 337.88,
> with more checking after version 344.11. I couldn't install version 319
> as it failed to build the Linux kernel module. I currently have 370.28 
> installed which supports both my GT610 and my GTX960.

`nvidia-smi -q` thinks it's running on bare hardware [when in VM]:
    GPU Virtualization Mode
        Virtualization mode         : None
Yet we know the driver refuses to load for Quadro 600 and your GTX.
(that two-faced bastard nVidia!) So there must be multiple checks.

> Maybe the next thing for me to try is to replicate your tests with a 
> FreeBSD VM.

Yes, give it a try. I'm about to give up, for nVidia card at least
(I was only using it for testing, as I didn't have AMD GPU handy
until recently).


[0]
# Host with idle FreeBSD VM guest running
# ubench 
Unix Benchmark Utility v.0.3
FreeBSD 10.3-STABLE FreeBSD 10.3-STABLE #0 r311343M: Thu Jan  5 02:31:50 PST 2017     xxx at yyy:/usr/obj/usr/src/sys/SORALX amd64
Ubench CPU:  2076542
Ubench MEM:  1296221
--------------------
Ubench AVG:  1686381

# Guest
# ubench
Unix Benchmark Utility v.0.3
FreeBSD 12.0-CURRENT FreeBSD 12.0-CURRENT #0 r311659: Tue Jan 10 14:18:44 PST 2017     xxx at zzz:/usr/obj/usr/src/sys/GENERIC amd64
Ubench CPU:  2053983
Ubench MEM:   489953
--------------------
Ubench AVG:  1271968


[1]
# Host
# diskinfo -tv /dev/ada0
/dev/ada0
        512             # sectorsize
        250059350016    # mediasize in bytes (233G)
        488397168       # mediasize in sectors
        4096            # stripesize
Seek times:
        Full stroke:      250 iter in   0.029146 sec =    0.117 msec
        Half stroke:      250 iter in   0.029950 sec =    0.120 msec
        Quarter stroke:   500 iter in   0.057272 sec =    0.115 msec
        Short forward:    400 iter in   0.045711 sec =    0.114 msec
        Short backward:   400 iter in   0.045748 sec =    0.114 msec
        Seq outer:       2048 iter in   0.072799 sec =    0.036 msec
        Seq inner:       2048 iter in   0.072256 sec =    0.035 msec
Transfer rates:
        outside:       102400 kbytes in   0.222412 sec =   460407 kbytes/sec
        middle:        102400 kbytes in   0.222026 sec =   461207 kbytes/sec
        inside:        102400 kbytes in   0.222842 sec =   459518 kbytes/sec

# Guest
# diskinfo -tv /dev/vtbd0
/dev/vtbd0
        512             # sectorsize
        22548644864     # mediasize in bytes (21G)
        44040322        # mediasize in sectors
        32768           # stripesize
Seek times:
        Full stroke:      250 iter in   0.112714 sec =    0.451 msec
        Half stroke:      250 iter in   0.100388 sec =    0.402 msec
        Quarter stroke:   500 iter in   0.132808 sec =    0.266 msec
        Short forward:    400 iter in   0.074504 sec =    0.186 msec
        Short backward:   400 iter in   0.096154 sec =    0.240 msec
        Seq outer:       2048 iter in   0.208239 sec =    0.102 msec
        Seq inner:       2048 iter in   0.061853 sec =    0.030 msec

Transfer rates:
        outside:       102400 kbytes in   0.201375 sec =   508504 kbytes/sec
        middle:        102400 kbytes in   0.196044 sec =   522332 kbytes/sec
        inside:        102400 kbytes in   0.197697 sec =   517964 kbytes/sec


-- 
[SorAlx]  ridin' VN2000 Classic LT