FreeBSD 12.1 i386 on Hyper-V Fails to Mount Root

[Wei Hu]

Copy Kib  who was the author of PTI and other relate patches.

This looks to be caused by Meltdown and Spectre mitigation patches which were 
checked into head couple years ago. See this link for details about these patches:
https://wiki.freebsd.org/SpeculativeExecutionVulnerabilities

The amd64  patches also caused problems
on Hyper-V (triple fault) when vmbus and other HyperV drivers (netvsc, storvsc etc)
were loaded separately. The workaround for this on amd64 is either build these
Hyper-V modules into the kernel or completely disable mitigation by
setting vm.pmap.pti="0" in /boot/loader.conf in the affect FreeBSD guests.

It looks the problem on i386 guests is different. While multiple patches have been 
checked in for amd64, there was only one patch r332489 for i386.  I am not sure
whether there is similar vm.pmap.pti option to disable the patch for i386 or not.

I am not familiar to these patches. 

@ Kib, will you be able to comment on this?

Thanks,
Wei



> -----Original Message-----
> From: Harry Schmalzbauer <freebsd at omnilan.de>
> Sent: Sunday, May 16, 2021 2:55 AM
> To: Wei Hu <weh at microsoft.com>; Chris Knight <stryqx at gmail.com>; freebsd-
> virtualization at freebsd.org
> Subject: Re: FreeBSD 12.1 i386 on Hyper-V Fails to Mount Root
> 
> Am 14.05.2021 um 18:44 schrieb Harry Schmalzbauer:
> > Am 16.10.2020 um 11:35 schrieb Wei Hu via freebsd-virtualization:
> >>> :
> >>>
> >> Did you try 12.0? Maybe something changed in 12.x causes failure.
> >> I386 is not well tested on Hyper-V. I will try to reproduce and take
> >> a look.
> >
> > The regression happened before 12.0.
> > Today I stumbled across this problem too and tried 11.4 (/May 8th,
> > 2020) /sucessfully, while 12.0 (/Oct 19th, 2018)/ is failing with i386
> > releases.
> >
> > One commit which is in 12 but not in 11.4 is r332489.
> >
> :
> :
> :
> > Unfortunately beyond my skills.
> 
> In sys/x86/x86/local_apic.c, I added this debug printf:
>          for (idx = IPI_DYN_FIRST; idx <= IPI_DYN_LAST; idx++) {
> printf("DEBUG: native_lapic_ipi_alloc(), idx = %i\n", idx);
>                  ip = &idt[idx];
>                  func = (ip->gd_hioffset << 16) | ip->gd_looffset;
>                  if ((!pti && func == (uintptr_t)&IDTVEC(rsvd)) ||
>                      (pti && func == (uintptr_t)&IDTVEC(rsvd_pti))) {
>                          vector = idx;
>                          setidt(vector, ipifunc, SDT_APIC, SEL_KPL, GSEL_APIC);
>                          break;
>                  }
>          }
> 
> This is my latest attempt to see if at least the origin of my fault tracing matches.
> It does.
> 
>  From #define IPI_DYN_FIRST   (APIC_IPI_INTS + 9) in sys/x86/include/apicvar.h
> I expected to see 252, 253 and 254, which I sucessfully verified.
> 
> Now comes the part I cannot read: (uintptr_t)&IDTVEC(rsvd)
> IDTVEC(rsvd) seems to be a assembler makro from sys/i386/i386/exception.s
> 
> But now it doesn't make any sense for me to dig further, I simply do not
> understand it.
> 
> Unfortunately having broken i386 support on Hyper-V is not a cosmetic issue.
> Removing "device hyperv" from kernel gives access to the v-hdd via legacy
> IDE/ada, but then I have no NIC.
> Choosing between missing HDD and missing network support is hard
> nowadays ;-)
> 
> Any help getting i386 up and running back on Hyper-V highly appreciated!  11.4
> is the latest working version, all i386-releases starting with 12.0 don't work!
> 
> Thanks,
> -harry