a bhyve and Windows Server 2012R2 story

[Evan Rowley]

I've booted my first Windows Server 2012R2 VM in bhyve. It took about
1 day's setup but I finally have it working. My notes below detail a
couple pitfalls and things which I did differently than described in
online references I used.

The further I progressed in this, the more I realized that the
directions I started from are based on directions from other places on
the net.
Started here: https://jameslodge.com/freebsd-bhyve-hypervisor-running-windows-server-2012-r2-standard/
Discovered this:
https://people.freebsd.org/~grehan/bhyve_uefi/windows_install.txt
And this: https://people.freebsd.org/~grehan/bhyve_uefi/windows_iso_repack.txt
Finally, saw this: https://wiki.freebsd.org/bhyve/Windows

Anyways, my machine is TRUEOS 11.0-CURRENTMAY2016
[root at true11] ~# uname -a
FreeBSD true11 11.0-CURRENTMAY2016 FreeBSD 11.0-CURRENTMAY2016 #19
5bab0d2(master): Fri May  6 17:56:25 UTC 2016
root at devastator:/usr/obj/usr/src/sys/GENERIC  amd64

CPU is AMD 6366 HE
Motherboard is SuperMicro H8DGU-F

In the guide, the p7zip-9.38.1_2 package is used. Here is the latest
version that I installed:

[root at true11] /mnt/vm0/ds0/iso# pkg info | grep 7z
p7zip-15.14                    File archiver with high compression ratio

Unfortunately, it seems this version of 7z was not able to extract the
UDF ISO correctly. Also, the version on this TrueOS11 machine required
a different syntax to specify the output folder, compared to the
syntax shown in the link above. Many error messages have been omitted,
but here is basically what happened:

[root at true11] /mnt/vm0/ds0/iso# //usr/local/bin/7z x
msdn/en_windows_server_2012_r2_with_update_x64_dvd_6052708.iso
-o{win2012iso}
....
ERROR: Data Error : support/logging/setupcletw.dll
ERROR: Data Error : support/logging/setupetw.dll
ERROR: Data Error : support/logging/setupugcetw.dll
ERROR: Data Error : support/logging/sysprepetw.dll
ERROR: Data Error : support/logging/windeployetw.dll
ERROR: Data Error : support/logging/winsetupetw.dll
ERROR: Data Error : support/samples/headlessunattend.xml

Sub items Errors: 2075

Archives with Errors: 1

Open Errors: 1

Sub items Errors: 2075


^ this is the output

For each of the files that "ERROR: Data Error" was shown, their z7
extracted contents were 0 bytes in size. Here is an example of the
last file's word count:

[root at true11] /mnt/vm0/ds0/iso# wc
win2012iso/support/samples/headlessunattend.xml
       0       0       0 win2012iso/support/samples/headlessunattend.xml

I decided to try extraction on Windows using graphical 7zip. The
version my computer had installed was 15.06. It worked. Here is the
word count for the same file, computed in Cygwin:

ejr at win7box1 ~
$ wc /cygdrive/c/Users/ejr/Downloads/Win2012R2/support/samples/headlessunattend.xml
 108  301 6475 /cygdrive/c/Users/ejr/Downloads/Win2012R2/support/samples/headlessunattend.xml

I also added NetKVM drivers from this version virtio-win-0.1.102.iso,
not virtio-win-0.1.96.iso as was specified in the guide.

After editing the set files on my Windows machine according to the
guide, I transferred the files back to my TrueOS 11 machine using
smbclient. The mkisofs command to create the new ISO worked
flawlessly.

I decided to use the 2016 version of the UEFI loader:
https://people.freebsd.org/~grehan/bhyve_uefi/BHYVE_UEFI_20160526.fd

Here is the command I used to boot the VM:

bhyve \
    -c 1 \
    -s 0,hostbridge \
    -s 3,ahci-hd,/dev/zvol/vm0/ds0/test/w2012R2-0/disk0 \
    -s 4,ahci-cd,/mnt/vm0/ds0/iso/Win2012R2.iso \
    -s 10,virtio-net,tap0 \
    -s 31,lpc -l com1,/dev/nmdm0A \
    -l bootrom,/mnt/vm0/ds0/test/w2012R2-0/BHYVE_UEFI_20160526.fd \
    -m 2G -H \
    -w w2012R2-0

My VM had a problem. It did boot and the SAC channels did start - but
somewhere after WIMApplyImage message was displayed in the SACSetupAct
- the VM would hang and top would show the bhyve process at 99% WCPU.
I destroyed the VM after it hanged like this for 1 hour. It took some
guessing, but eventually I recalled reading somewhere - possibly on a
KVM/QEMU/Xen wiki page - that IOMMU is needed for more than 1 core. I
suspected that this machine's IOMMU was disabled because I've set the
SuperMicro BIOS to Optimized Defaults a couple times before and
noticed that IOMMU would get turned off. I couldn't recall if I had
re-enabled it. Rather than reboot the machine and set IOMMU to
Enabled, I decided to re-launch the bhyve VM with only 1 core. This
way I was able to compete the installation. Later, enabling IOMMU and
relaunching the VM with 2 cores worked.

I had difficulty at first with the network configuration. TrueOS has
ipfw enabled and I suspect it was originally blocking my RDP traffic.
Also, I did not have any successful network connections before I set
and loaded what's described in 21.7.7 Persistent Configuration:
https://www.freebsd.org/doc/handbook/virtualization-host-bhyve.html

At first, I attempted to RDP to an IP I had assigned from the SAC
channel using the i <#> <ip> <netmask> <gateway> command. This never
worked and I am not sure why. Eventually, I created a CMD channel,
then used ipconfig to determine that Windows had leased a different IP
via DHCP. Using RDP to reach the DHCP IP worked! Oddly, I did not see
the static IP I had set in SAC updated with the DHCP one - but I
checked later, and it was updated. I'm new to SAC and maybe this is
just one of it's quirks. Not a FreeBSD problem in any case, but
perhaps a stumbling block others on freebsd-virtualization may
encounter.


-- 
 - EJR