Windows 10 guests fail to boot when attempting to passthrough network card

[Rodney W. Grimes]

> Does the card have multiple BARs? Can you send us the output of pciconf -lvb for the card?
> 
> There is a bug in bhyve where if the OS assigns the 0xffffffff sentinel to more than one BAR simultaneously it is not compatible with bhyve?s memory model for the guest.

The "we talked" below is myself and Nick talking about exactly that
which came from jhb@ pointing out to me in a hall wall
conversation here at BSDCan that this is the cause of this
problem and is a known issue.

Does anyone have fixes for this issue?

> 
> On Sat, May 18, 2019 at 21:14, Nick Principe <nap at ixsystems.com> wrote:
> 
> > Yeah, that triggers a barf with error 2:
> > mmio_rb_lookup: barf ENOENTerror is 2
> > mr.name is passthru-pci-5
> > mr.flags is 0
> > mr.arg2 is 0
> > mr.base is 18446744073709027328
> > mr.size is 524288
> > Assertion failed: (error == 0), function modify_bar_registration, file
> > /usr/src/usr.sbin/bhyve/pci_emul.c, line 510.
> >
> > Interestingly enough, after we talked I simply commented out the assert at
> > 504 in pci_emul.c and Windows booted fine and can see the Chelsio card with
> > both interfaces. I'm not sure if it would fall over once I started actually
> > using it or not.
> >
> > `nap
> >
> > On Sat, May 18, 2019 at 4:06 PM Rodney W. Grimes <
> > freebsd-rwg at gndrsh.dnsmgr.net> wrote:
> >
> >> > I have noticed that Windows 10 guests fail to boot when attempting to
> >> pass
> >> > through a network card. I believe I have observed this with both cxgbe
> >> > (t580) and mlx5en cards, but only have a cxgbe to test with now. Without
> >> > passthrough, the Windows 10 guest boots and operates normally.
> >> >
> >> > FreeBSD guests (12.0-RELEASE) have no issue when booting with the cxgbe
> >> > card passed through - I can kldload cxgbe and I get both cxl ports
> >> showing
> >> > up in the FreeBSD guest.
> >> >
> >> > I have tested this with both 12.0-RELEASE and head (13-CURRENT r347883)
> >> as
> >> > the host OS with no change in behavior. The bhyve output is:
> >> > Unhandled ps2 keyboard command 0x02
> >> > Unhandled ps2 keyboard command 0x02
> >> > Assertion failed: (error == 0), function modify_bar_registration, file
> >> > /usr/src/usr.sbin/bhyve/pci_emul.c, line 504.
> >> > fbuf frame buffer base: 0x943600000 [sz 16777216]
> >> >
> >> > Two main suggestions from discussions at BSDCan this week were:
> >> > - Capture pciconf -lvb from the FreeBSD guest
> >> > - Add some printf to pci_emul.c to capture some values when there is an
> >> > error
> >> >
> >> > I've captured the above, and a lot of other relevant info, in a Google
> >> Doc
> >> > here (too big to post directly):
> >> >
> >> https://docs.google.com/document/d/1t-UVIO9Aq0TPUFHyo1nVscqaW1LoPuNhfLPitL8oeTs/edit?usp=sharing
> >> >
> >> > `nap
> >>
> >> To confirm what I see from looking at your data could you tell me if this
> >> patch triggers a barf?
> >>
> >> --- mem.c.orig 2019-05-18 20:04:26.707995000 +0000
> >> +++ mem.c 2019-05-18 20:04:02.205119000 +0000
> >> @@ -97,6 +97,7 @@
> >> return (0);
> >> }
> >>
> >> +printf("mmio_rb_lookup: barf ENOENT");
> >> return (ENOENT);
> >> }
> >>
> >> --
> >> Rod Grimes
> >> rgrimes at freebsd.org
> >>
> >
> > --
> > Nick Principe
> > Performance Engineering Supervisor
> > iXsystems, Inc.
> > Ph: (408) 943-4100 x341
> > Fx: (408) 943-4101
> > _______________________________________________
> > freebsd-virtualization at freebsd.org mailing list
> > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> > To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe at freebsd.org"
-- 
Rod Grimes                                                 rgrimes at freebsd.org