KVM+FreeBSD+82599 10G NIC doesn't work.
GemaEvor
legumen at foxmail.com
Fri May 17 03:01:48 UTC 2013
Thanks a lot Jack.
According to my test, SRIOV works fine!
It's not the way we want, but this solution really helps.
So, we'll spend more time to solve the real problem, while we can use SRIOV as a substitution.
------------------
Best wishes.
Jason Shi
------------------ Original ------------------
From: "Jack Vogel"<jfvogel at gmail.com>;
Date: Thu, May 16, 2013 04:02 PM
To: "GemaEvor"<legumen at foxmail.com>;
Cc: "freebsd-virtualization"<freebsd-virtualization at freebsd.org>;
Subject: Re: KVM+FreeBSD+82599 10G NIC doesn't work.
The environment that I tested in when the virtualization support was put intothe driver was using SRIOV, not pass-through, we have never validated the
latter, and whenever it crops up it seems to be having problems.
I can put it into my queue to look into, but I can't say how quickly I can get
to it, very busy here.
Jack
On Thu, May 16, 2013 at 12:45 AM, GemaEvor <legumen at foxmail.com> wrote:
Hi gurus:
I'm working on a FreeBSD virtulizaion project under KVM.
Everything works fine, except the 10G Intel 82599 SFP NIC.
I'm sure there are some body else who have encountered this problem before. See:
FreeBSD guest with VTD NIC not passing traffic
http://comments.gmane.org/gmane.comp.emulators.kvm.devel/83806
VT-d not working for FreeBSD 9.0 guest
http://permalink.gmane.org/gmane.comp.emulators.kvm.devel/91074
But, unfortunately, none of them solved it or gave the clearly way to solve it. :<
Firstly, let's make it clearly. This problem could not be reproduced with:
1. Xen hypervisor + FreeBSD guest os
2. or KVM + Linux guest os (e.g. Fedora 12)
So, it must be something wrong with KVM or FreeBSD.
Here is my environment:
Host OS:
OpenSUSE 12.2
Linux linux-kj2u 3.4.6-2.10-desktop #1 SMP PREEMPT Thu Jul 26 09:36:26 UTC 2012 (641c197) x86_64 x86_64 x86_64 GNU/Linux
QEMU emulator version 1.1.1 (kvm-1.1.1-1.8.1), Copyright (c) 2003-2008 Fabrice Bellard
Guest OS:
FreeBSD 9.1 amd64
What I did:
1.Make the pci device of 82599SFP assignale:
[CODE]
0d:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
Subsystem: QLogic, Corp. Device 00e4
Flags: fast devsel, IRQ 25
Memory at f8fe0000 (64-bit, prefetchable) [size=128K]
I/O ports at cc00 [size=32]
Memory at f8fdc000 (64-bit, prefetchable) [size=16K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Capabilities: [70] MSI-X: Enable- Count=64 Masked-
Capabilities: [a0] Express Endpoint, MSI 00
Capabilities: [e0] Vital Product Data
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Device Serial Number 00-00-00-ff-ff-00-00-00
Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
Kernel driver in use: pci-stub
[/CODE]
2. Passthrough it to the guest FreeBSD os. Here is my vm configuration file
[FILE]
<domain type='kvm'>
<name>FreeBSD9.1</name>
<uuid>4d56613e-400c-f37d-2a9a-e0332d81ca63</uuid>
<memory>2097152</memory>
<currentMemory>2097152</currentMemory>
<vcpu>2</vcpu>
<os>
<type arch='x86_64' machine='pc'>hvm</type>
<boot dev='hd'/>
</os>
<features>
<pae/>
<apic/>
<acpi/>
</features>
<clock offset='utc'>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<emulator>/usr/bin/qemu-kvm</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='raw' cache='default'/>
<source file='/home/vm/FreeBSD9.1/FreeBSD9.1.raw'/>
<target dev='hdc'/>
</disk>
<interface type='bridge'>
<source bridge='br0'/>
<mac address='52:54:00:0d:92:6f'/>
<model type='rtl8139'/>
</interface>
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x0d' slot='0x00' function='0x0'/>
</source>
</hostdev>
<input type='mouse' bus='ps2'/>
<graphics type='vnc' port='-1' autoport='yes'/>
</devices>
</domain>
[/FILE]
3.The device could be probed and attached by FreeBSD:
[CODE]
ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.4.8> port 0xc020-0xc03f mem 0xfe000000-0xfe01ffff,0xfe020000-0xfe023fff irq 11 at device 4.0 on pci0
ix0: Using MSIX interrupts with 2 vectors
ix0: Ethernet address: 00:0c:bd:05:7d:4a
ix0: PCI Express Bus: Speed 5.0Gb/s Width x8
[/CODE]
But no interrupt could be found:
[CODE]
interrupt total rate
irq1: atkbd0 143 0
irq4: uart0 177 1
irq15: ata1 1593 10
cpu0:timer 7140 46
Total 9053 58
[/CODE]
And ix0 keeps to be "no carrier" (even if I config an IP address on it and ping):
[CODE]
ix0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=401bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
ether 00:0c:bd:05:7d:4a
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: no carrier
[/CODE]
4.In the host side, kvm could see the change made by the guest on the device, but no irq:
[CODE]
linux-kj2u:~ # cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 CPU8 CPU9 CPU10 CPU11 CPU12 CPU13 CPU14 CPU15 CPU16 CPU17 CPU18 CPU19 CPU20 CPU21 CPU22 CPU23
210: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge kvm:0000:0d:00.0
211: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge kvm:0000:0d:00.0
[/CODE]
So, anyone who has the same problem or has a clue to solve it, or has some ideas to do a debug job, please help.
Thanks!
_______________________________________________
freebsd-virtualization at freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe at freebsd.org"
More information about the freebsd-virtualization
mailing list