VMX exit reason=33 and general userboot.so questions

[Anish]

>idtr[0]         0x0000000000000000/0x00000000

IDTR is not set, you can set the value same as in GDTR, from libvmm

       /* GDTR, IDTR */
        desc_base = 0;
        desc_limit = 0xffff;
        desc_access = 0;
        error = vm_set_desc(vmctx, vcpu, VM_REG_GUEST_GDTR,
                            desc_base, desc_limit, desc_access);
        if (error != 0)
                goto done;

        error = vm_set_desc(vmctx, vcpu, VM_REG_GUEST_IDTR,
                            desc_base, desc_limit, desc_access);

>I’m assuming entry_ctls[0] = 0x91fb is the VM-entry control,

Yes, it is VM entry control. Its value need to be determined based on
MSR_VMX_ENTRY_CTLS
and MSR_VMX_TRUE_ENTRY_CTLS  MSRs, see

        /* Check support for VM-entry controls */
        error = vmx_set_ctlreg(MSR_VMX_ENTRY_CTLS, MSR_VMX_TRUE_ENTRY_CTLS,
            VM_ENTRY_CTLS_ONE_SETTING, VM_ENTRY_CTLS_ZERO_SETTING,
            &entry_ctls);

-Anish


On Wed, Feb 21, 2018 at 11:08 AM, Fabian Freyer <
fabian.freyer at physik.tu-berlin.de> wrote:

> Hi Peter,
>
> thanks for your response!
>
> On 21 Feb 2018, at 17:59, Peter Grehan wrote:
> >  >        exit_reason     33
> >
> >  From the Intel SDM, vol 3B Appendix C, this error is "VM-entry failure
> due to invalid guest state".
>
> Yes, I’m currently going through SDM, vol 3B, section 26.3, and
>
> >  These errors can be difficult to debug given the large amount of guest
> state involved :(
>
> definitely accurately describes the experience. That said, I’m going
> through each check and trying to see what is going wrong.
>
> >  However, looking at the state from your dump:
> >
> >  > tr desc[0]      0x0000000000000000/0x00000000/0x00000000
> >
> >  I believe you will have to set this. Here's the comment and relevant
> code fragment from grub2-bhyve grub-core/kern/emu/bhyve_
> hostif.c:grub_emu_bhyve_boot32()
> >
> >   /*
> >    * XXX TR is pointing to null selector even though we set the
> >    * TSS segment to be usable with a base address and limit of 0.
> >    * Has to be 8b or vmenter will fail
> >    */
> >   desc_access = 0x0000008b;
> >   assert(vm_set_desc(bhyve_ctx, 0, VM_REG_GUEST_TR, 0x1000, 0x67,
> >                      desc_access) == 0);
> >
> >  grub2-bhyve has been able to load/boot multiboot images, so I suspect
> the register settings in grub_emu_bhyve_boot32() are a good place to start
> from.
>
> Thanks. I’ve tried setting all registers (especially) the segment
> registers as grub2-bhyve does, but I still don’t seem to be able to boot
> correctly. Fixing the TR descriptor didn’t do it.
>
> Here’s a list of the descriptors I set up:
>
> ds desc[0]      0x0000000000000000/0xffffffff/0x00000093
> es desc[0]      0x0000000000000000/0xffffffff/0x00000093
> fs desc[0]      0x0000000000000000/0xffffffff/0x00000093
> gs desc[0]      0x0000000000000000/0xffffffff/0x00000093
> ss desc[0]      0x0000000000000000/0xffffffff/0x00000093
> cs desc[0]      0x0000000000000000/0xffffffff/0x0000009b
> tr desc[0]      0x0000000000001000/0x00000067/0x0000008b
> ldtr desc[0]    0x0000000000000000/0x0000ffff/0x00010082
>
>
> gdtr points to a default gdt as set up in grub2-bhyve:
>
> static uint16_t bhyve_gdt[] = {
>   0x0000, 0x0000, 0x0000, 0x0000,       /* Null */
>   0x0000, 0x0000, 0x0000, 0x0000,       /* Null #2 */
>   0xffff, 0x0000, 0x9a00, 0x00cf,       /* code */
>   0xffff, 0x0000, 0x9200, 0x00cf,       /* data */
>   0x0000, 0x0000, 0x8900, 0x0080        /* tss */
> };
>
> gdtr[0]         0x0000000000100161/0x00000027
> idtr[0]         0x0000000000000000/0x00000000
>
> The rest is set up as by grub2-bhyve:
> cs[0]           0x0008
> ds[0]           0x0010
> es[0]           0x0010
> fs[0]           0x0010
> gs[0]           0x0010
> ss[0]           0x0010
> tr[0]           0x0000
> ldtr[0]         0x0000
>
> I’m assuming entry_ctls[0] = 0x91fb is the VM-entry control, but is it the
> value described by Table 24-7 of the SDM Volume 3B? Is this constant for
> the first entry after running bhyveload?  That is, when writing some
> compliance checks against section 26.3.1, can I just hardcode that value?
>
> I’m starting to dig through the bhyverun code, but I’m still pretty new to
> VMX stuff, so I am ending up with more questions than answers.
>
> Fabian
>