KVM+FreeBSD+82599 10G NIC doesn't work.

Thu May 16 07:45:21 UTC 2013

Hi gurus:

I'm working on a FreeBSD virtulizaion project under KVM.
Everything works fine, except the 10G Intel 82599 SFP NIC.

I'm sure there are some body else who have encountered this problem before. See:
FreeBSD guest with VTD NIC not passing traffic
http://comments.gmane.org/gmane.comp.emulators.kvm.devel/83806
VT-d not working for FreeBSD 9.0 guest
http://permalink.gmane.org/gmane.comp.emulators.kvm.devel/91074

But, unfortunately, none of them solved it or gave the clearly way to solve it. :<

Firstly, let's make it clearly. This problem could not be reproduced with:
1. Xen hypervisor + FreeBSD guest os
2. or KVM + Linux guest os (e.g. Fedora 12)
So, it must be something wrong with KVM or FreeBSD.

Here is my environment:
Host OS:
OpenSUSE 12.2
Linux linux-kj2u 3.4.6-2.10-desktop #1 SMP PREEMPT Thu Jul 26 09:36:26 UTC 2012 (641c197) x86_64 x86_64 x86_64 GNU/Linux
QEMU emulator version 1.1.1 (kvm-1.1.1-1.8.1), Copyright (c) 2003-2008 Fabrice Bellard

Guest OS:
FreeBSD 9.1 amd64

What I did:
1.Make the pci device of 82599SFP assignale:
[CODE]
0d:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
        Subsystem: QLogic, Corp. Device 00e4
        Flags: fast devsel, IRQ 25
        Memory at f8fe0000 (64-bit, prefetchable) [size=128K]
        I/O ports at cc00 [size=32]
        Memory at f8fdc000 (64-bit, prefetchable) [size=16K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
        Capabilities: [70] MSI-X: Enable- Count=64 Masked-
        Capabilities: [a0] Express Endpoint, MSI 00
        Capabilities: [e0] Vital Product Data
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [140] Device Serial Number 00-00-00-ff-ff-00-00-00
        Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
        Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
        Kernel driver in use: pci-stub
[/CODE]

2. Passthrough it to the guest FreeBSD os. Here is my vm configuration file
[FILE]
<domain type='kvm'>
  <name>FreeBSD9.1</name>
  <uuid>4d56613e-400c-f37d-2a9a-e0332d81ca63</uuid>
  <memory>2097152</memory>
  <currentMemory>2097152</currentMemory>
  <vcpu>2</vcpu>
  <os>
    <type arch='x86_64' machine='pc'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <pae/>
    <apic/>
    <acpi/>
  </features>
    <clock offset='utc'>
    </clock>
    <on_poweroff>destroy</on_poweroff>
    <on_reboot>restart</on_reboot>
    <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-kvm</emulator>
      <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='default'/>
      <source file='/home/vm/FreeBSD9.1/FreeBSD9.1.raw'/>
      <target dev='hdc'/>
    </disk>
      <interface type='bridge'>
        <source bridge='br0'/>
        <mac address='52:54:00:0d:92:6f'/>
        <model type='rtl8139'/>
      </interface>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x0d' slot='0x00' function='0x0'/>
      </source>
    </hostdev>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes'/>

  </devices>
</domain>
[/FILE]

3.The device could be probed and attached by FreeBSD:
[CODE]
ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.4.8> port 0xc020-0xc03f mem 0xfe000000-0xfe01ffff,0xfe020000-0xfe023fff irq 11 at device 4.0 on pci0
ix0: Using MSIX interrupts with 2 vectors
ix0: Ethernet address: 00:0c:bd:05:7d:4a
ix0: PCI Express Bus: Speed 5.0Gb/s Width x8
[/CODE]

But no interrupt could be found:
[CODE]
interrupt                          total       rate
irq1: atkbd0                         143          0
irq4: uart0                          177          1
irq15: ata1                         1593         10
cpu0:timer                          7140         46
Total                               9053         58
[/CODE]

And ix0 keeps to be "no carrier" (even if I config an IP address on it and ping):
[CODE]
ix0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=401bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
	ether 00:0c:bd:05:7d:4a
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: no carrier
[/CODE]

4.In the host side, kvm could see the change made by the guest on the device, but no irq:
[CODE]
linux-kj2u:~ # cat /proc/interrupts 
            CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7       CPU8       CPU9       CPU10      CPU11      CPU12      CPU13      CPU14      CPU15      CPU16      CPU17      CPU18      CPU19      CPU20      CPU21      CPU22      CPU23      
 210:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      kvm:0000:0d:00.0
 211:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      kvm:0000:0d:00.0
[/CODE]

So, anyone who has the same problem or has a clue to solve it, or has some ideas to do a debug job, please help. 

Thanks!