VT-d issues with FreeBSD 9.0 guest using 82599 HW

[Shesha Sreenivasamurthy]

Hi,
Earlier I had sent the same email to freebsd general discussion. However, I
felt this forum is more appropriate. I apologize for double email.

I have an issue with enabling VT-d with freebsd guest and was wondering of
anyone out there are experiencing the similar issues.

I am using the following setup
 HOST: Centos 6.2
 Hypervisor: KVM
 Hardware: ixgbe 82599EB (10 GB nic)
 Guest: freebsd 9.0
 Guest Driver version 2.4.4

When I use MSI/MSIX the link status is shown as "no carrier". When I
disable MSI and MSIX in the guest so that it can fall back on INTx, I see
that the link becomes active. However not packets are sent or received.

When pinged from outside, sysctl shows

dev.ix.0.queue0.txd_head: 0
dev.ix.0.queue0.rxd_head: 0
dev.ix.0.queue0.rxd_tail: 2047
dev.ix.0.queue0.rx_packets: 0
dev.ix.0.mac_stats.good_octets_rcvd: 320
dev.ix.0.mac_stats.good_pkts_rcvd: 5

When pinged from inside, sysctl shows

dev.ix.0.queue0.txd_head: 0
dev.ix.0.queue0.rxd_head: 0
dev.ix.0.queue0.txd_tail: 4
dev.ix.0.queue0.rxd_tail: 2047
dev.ix.0.queue0.tx_packets: 4
dev.ix.0.queue0.rx_packets: 0
dev.ix.0.mac_stats.good_octets_rcvd: 320
dev.ix.0.mac_stats.good_pkts_rcvd: 5
dev.ix.0.mac_stats.good_octets_txd: 0
dev.ix.0.mac_stats.good_pkts_txd: 0

I guess, hardware is getting the packets but not put in the ringbuffer.

If I install freebsd 9.0 on bare metel and have the same hardware and same
driver running, the link comes up and packets are transferred. Therefore,
in case of VT-d (PCI passthrough) the driver is not setting up things
correctly.

Another data point is, if Centos guests is used instead of freebsd guest,
the hardware comes up correctly and can send/receive packets. The freebsd
driver that I am using is the one posted on the intel's website.

http://downloadcenter.intel.com/Detail_Desc.aspx?agr=Y&ProdId=3189&DwnldID=14688&ProductFamily=Ethernet+Components&ProductLine=Ethernet+Controllers&ProductProduct=Intel%C2%AE+82599+10+Gigabit+Ethernet+Controller&DownloadType=Drivers&OSFullname=FreeBSD*eng