[GPU pass-through] no compatible bridge window for claimed BAR

[Alex Erley]

Hello Peter,

Many thanks for your insight!

Changing PCI_EMUL_MEMBASE64
  from 0xD000000000 to 0x0440000000 (16Gb guest memory limit)
and PCI_EMUL_MEMLIMIT64
  from 0xFD00000000 to 0x07ffffffff (32Gb host memory limit)
still fails.
 From guest dmesg:
  ...
  [mem 0xc0000000-0xffffffff] available for PCI devices
  ...
  pci 0000:00:01.0: [10de:1c03] type 00 class 0x030000
  pci 0000:00:01.0: reg 0x10: [mem 0xc0000000-0xc0ffffff]
  pci 0000:00:01.0: reg 0x14: [mem 0x440000000-0x44fffffff 64bit pref]
  pci 0000:00:01.0: reg 0x1c: [mem 0xc2000000-0xc3ffffff 64bit pref]
  pci 0000:00:01.0: reg 0x24: [io  0x2000-0x207f]
  pci 0000:00:01.0: reg 0x30: [mem 0xd3000000-0xd307ffff pref]
  pci 0000:00:01.1: [10de:10f1] type 00 class 0x040300
  pci 0000:00:01.1: reg 0x10: [mem 0xc4000000-0xc4003fff]
  ...
  pci 0000:00:01.0: can't claim BAR 1 [mem 0x440000000-0x44fffffff 64bit
pref]: no compatible bridge window
  ...
  pci 0000:00:01.0: BAR 1: no space for [mem size 0x10000000 64bit pref]
  pci 0000:00:01.0: BAR 1: trying firmware assignment [mem
0x440000000-0x44fffffff 64bit pref]
  pci 0000:00:01.0: BAR 1: assigned [mem 0x440000000-0x44fffffff 64bit pref]

It seems guest requires BAR to be within its addressable space.
Changing PCI_EMUL_MEMBASE64 to 0x00c0000000 (as guest suggests in its
dmesg) didn't work either:

   bhyve: failed to initialize BARs for PCI 1/0/0
   device emulation initialization error: Cannot allocate memory

 From the BHyve sources it becomes evident that when claimed BAR size
is <= 32Mb, it is mandatory allocated as 32-bit BAR.
The only case BAR goes for 64-bit allocation is when its size > 32Mb.
In my case it is 256 Mb and it fails only for this BAR.

I think the 64-bit BAR allocation still has to be fixed in BHyve code.
Anyway, we are getting closer :)
Any ideas are really appreciated.

Have a nice day,
Alex

On 3/12/20 1:31 AM, Peter Grehan wrote:
> Hi Alex,
> 
>>> dmesg | grep "no compatible bridge window"
>> pci 0000:00:01.0: can't claim BAR 1 [mem 0xd000000000-0xd00fffffff
>> 64bit pref]: no compatible bridge window
> ...>  From what I can read from all the info above,
>> somehow the requested memory space for BAR1 for that device is 
>> 0xd000000000-0xd00fffffff which is out of addressable space on the
>>  system.
> 
> Yep, that's the issue, and it's a bhyve bug - there is no check to
> see if the 64-bit window is within the addressable range of the
> processor.
> 
> A quick fix is to change the constant for that range in pci_emul.c
> 
> #define PCI_EMUL_MEMBASE64      0xD000000000UL #define
> PCI_EMUL_MEMLIMIT64     0xFD00000000UL
> 
> .. to a value that is within the address bits of the CPU, but also 
> above guest DRAM.
> 
> later,
> 
> Peter.