kern/184059: Mps SCSI driver causes FreeBSD to hang during boot when running as a virtual machine.

[Robert Sevat]

>Number:         184059
>Category:       kern
>Synopsis:       Mps SCSI driver causes FreeBSD to hang during boot when running as a virtual machine.
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon Nov 18 20:30:00 UTC 2013
>Closed-Date:
>Last-Modified:
>Originator:     Robert Sevat
>Release:        FreeBSD 9.2 and FreeBSD 10.0 Beta 3
>Organization:
Student
>Environment:
root at fbsdtest:~ # uname -a
FreeBSD fbsdtest 10.0-BETA3 FreeBSD 10.0-BETA3 #0: Mon Nov 18 19:35:35 CET 2013
root at fbsdtest:/usr/obj/usr/src/sys/GENERIC_Custom5 amd64

I have also tried a default amd 64 fbsd 9.2.
>Description:
Greetings,

I have a problem with forwarding an LSI 2308 via vt-d in KVM to a FreeBSD virtual machine. FreeBSD (9.2 and 10.0 beta 3) will hang during the boot.

Hardware Setup:

Supermicro X10SL7-F with LSI 2308 flashed to IT mode
8x4 GB ecc ram
Haswell Xeon E3-1230V3

Software Setup:
Ubuntu 12.04.3 LTS 64 bit + latest KVM version.

uname -a
Linux Secretum 3.8.0-33-generic #48~precise1-Ubuntu SMP Thu Oct 24 16:28:06 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

dpkg --list | grep kvm 
ii  kvm          1:84+dfsg-0ubuntu16+1.0+noroms+0ubuntu14.12 dummy transitional package from kvm to qemu-kvm
ii  kvm-ipxe     1.0.0+git-3.55f6c88-0ubuntu1                PXE ROM's for KVM
ii  qemu-kvm     1.0+noroms-0ubuntu14.12                     Full virtualization on i386 and amd64 hardware



Under KVM I have the following 3 virtual machines installed, I have tried forwarding the LSI 2308 to all three virtual machines. It works perfectly under Ubuntu, but both FreeBSD vms will hang during the boot.

FreeBSD 9.2
FreeBSD 10.0 beta 3
FreeBSD 10.0 live cd
Ubuntu 12.04 LTS

If I run FreeBSD 10.0 beta 3 directly on the hardware, it does recognize the raid controller and it'll use the mps0 driver. Everything works fine then.

So the problem is that for some reason FreeBSD hangs during boot if you forward the LSI 2308 via vt-d, and I have no idea why.

It will hang and give the following error:

http://i.imgur.com/hAMxwR7.png
http://i.imgur.com/rKALeXZ.png

While doing so the FreeBSD virtual machine uses 300% cpu, so it maxes out 3 cores. And it will stay like that.

After googling a bit some people suggested turning off msi / msix in the loader.conf

hw.pci.enable_msi="0"
hw.pci.enable_msix="0"

I have tried this on both freebsd virtual machines, it makes no difference. It still hangs.


After having tried all this a collegue suggested trying to turn off drivers in the kernel. After compiling a few kernels and seeing if I could boot I managed to trace the root of the problem to the mps SCSI driver.

If I compile a kernel without that driver and pass through the raid controller via vt-d, the freebsd virtual machine will no longer hang during boot. I obviously also don't have a driver for my raid controller then..

So something in the mps driver is causing the deadlock which causes FreeBSD to hang / lockup during boot.

I can give you more information if necessarry.


Kind Regards

Robert Sevat

>How-To-Repeat:
Install FreeBSD as a virtual machine under KVM and pass through a LSI 2308 raid controller via vt-d. This will cause the mps driver to lock up during boot.
>Fix:
Change the mps SCSI driver.

>Release-Note:
>Audit-Trail:
>Unformatted: