iommu must be enabled for PVH hardware domain

[Ruslan Makhmatkhanov]

Roger Pau Monné wrote on 05/13/15 11:14:
> Hello,
>
> El 13/05/15 a les 10.04, Ruslan Makhmatkhanov ha escrit:
>> Roger Pau Monné wrote on 05/12/15 18:50:
>>> Hello,
>>>
>>> El 12/05/15 a les 17.20, Ruslan Makhmatkhanov ha escrit:
>>>> Hello,
>>>>
>>>> we have machine with VT-d and EPT-capable CPU:
>>>> Intel(R) Xeon(R) CPU E5504 [1]
>>>>
>>>> The system is FreeBSD 11.0-CURRENT r282694 installed from
>>>> ftp.freebsd.org iso.
>>>>
>>>> But dom0 fails to boot at this hardware with "iommu must be enabled for
>>>> PVH hardware domain" [2]. VT-d support is indeed not detected at kernel
>>>> level. Here is dmesg with VT-d enabled in BIOS [3] and dmesg with VT-d
>>>> disabled in BIOS [4]. It's look like there is no difference on both
>>>> outputs. Here is the proofpic of that the system is VT-d capable and the
>>>> support is enabled in BIOS [5]. Here is acpidump -t output [6]
>>>
>>> My bet right now is be that Xen detects that your IOMMU contains erratas
>>> and disables it. Can you add "iommu=debug,force" to your xen_cmdline and
>>> try again?
>>>
>>> Also, can you paste the contents of your /boot/loader.conf?
>>>
>>> Roger.
>>
>> You are right:
>>
>> [VT-D] Disabling IOMMU due to Intel 5500/5520/X58 Chipset errata #47, #53
>>
>> and after that:
>>
>> ----------------------------------------------
>> Panic on CPU 0:
>> Couldn't enable IOMMU and iommu=required/force
>> ----------------------------------------------
>>
>> Reboot in five seconds.
>>
>> But what this means practically? Some chipset bugs preventing us to use
>> this feature. Can this circumvented somehow? Thanks.
>
> AFAIK iommu=force should force the usage of the IOMMU even if erratas
> are detected. You can also try with iommu=force,no-intremap.
>
> Roger.


Yes, the system is now booted successfully! But `xl list` doesn't work with:

root@:~ # xl list
libxl: error: libxl.c:669:libxl_list_domain: getting domain info list: 
Cannot allocate memory
libxl_list_domain failed.

I googled for this error, and make ktrace/kdump on `xl list`. Here is 
the output: https://dpaste.de/mAyX/raw

It referencing many shared libraries, that is not available on this 
system. Dunno if this is cause of problem.

And here is the `xl info` and `xl dmesg`: https://dpaste.de/H1dV/raw

Max wired pages is unlimited:

root@:~ # sysctl vm.max_wired
vm.max_wired: -1

Also tried to set: vm.max_wired=1048576, but error is the same.

Shell memory values are unlimited too:

root@:~ # ulimit -a
cpu time               (seconds, -t)  unlimited
file size           (512-blocks, -f)  unlimited
data seg size           (kbytes, -d)  33554432
stack size              (kbytes, -s)  524288
core file size      (512-blocks, -c)  unlimited
max memory size         (kbytes, -m)  unlimited
locked memory           (kbytes, -l)  64
max user processes              (-u)  6656
open files                      (-n)  57987
virtual mem size        (kbytes, -v)  unlimited
swap limit              (kbytes, -w)  unlimited
sbsize                   (bytes, -b)  unlimited
pseudo-terminals                (-p)  unlimited
kqueues                         (-k)  unlimited

The same error when we trying to create new guest domain:
http://dpaste.com/1PD63M9

Domain configuration is taken here:
http://wiki.xen.org/wiki/FreeBSD_Dom0

Can anything be done to solve this? Thank you.

-- 
Regards,
Ruslan

T.O.S. Of Reality