Testing VF/PF code

Tue May 29 18:40:51 UTC 2018

Here are the steps that we use. If you just want to use bare metal, I think you’ll need to use passthrough : false; in the .conf.
·         The only supported output from virtual machines is serial console (i.e. you don't get a nice GUI if you use a Linux guest)
·         You should start your VM in a tmux session, so you can disconnect from the VM without having to shut it down.
·         To use SR-IOV, it's required that your host's FreeBSD version is 11 or newer, but it's recommended that your guest is 11 or newer, too.
·         A listing of currently active VMs can be found in /dev/vmm
·         VM names can only be up to 31 characters long
Steps to start a VM
1.    If you want to attach VF's to the virtual machine, you'll need to run these two commands first:
kenv hw.vmm.force_iommu=1
kldload vmm
2.    Then you will want to set up a tap interface so that your VM has network connectivity (either to the outside or your host, but in this case, we'll use a bridge so it has outside connectivity).
ifconfig tap0 create
sysctl net.link.tap.up_on_open=1
ifconfig bridge0 create
ifconfig bridge0 addm igb0 addm tap0
ifconfig bridge0 up
3.    Create a disk image that will serve as your VM's hard disk
truncate -s 40g freebsdvm-fs.img
4.    Booting up a VM is a two-step process: start the bootloader using bhyveload (only if you're using a FreeBSD guest), and then boot the kernel using the bhyve command.
·         Example bhyveload+bhyve commands I used to start a FreeBSD guest installation (where disc1.iso is a FreeBSD install disc .iso):

bhyveload -m 4096M -d disc1.iso fbsdvm

bhyve -A -H -P -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap0 -s 3:0,virtio-blk,freebsdvm-fs.img -s 4:0,ahci-cd,disc1.iso -l com1,stdio -c 4 -m 4096M fbsdvm
·         The -m arguments given to both commands need to match!
·         Specify the xterm option when prompted during the install process to get color.
·         Letting the installer configure partitions for you is fine.
·         Then, the following will run your new, installed FreeBSD VM:

bhyveload -m 4096M -d freebsdvm-fs.img fbsdvm

bhyve -A -H -P -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap0 -s 3:0,virtio-blk,freebsdvm-fs.img -l com1,stdio -c 4 -m 4096M fbsdvm

Using VFs (SR-IOV)
1.    kldload your SR-IOV enabled PF driver if it isn't already loaded
2.    Create a .conf (for iovctl to use; the file name doesn't matter, but it's recommended to use the interface name) for each PF interface that you want to add VFs to.
·         An example for an ixl interface:

·         PF {

·             device : "ixl0";

·             num_vfs : 3;

·         }

·         DEFAULT {

·             passthrough : true;

·         }

·         VF-0 {

·             mac-addr : "02:01:02:03:04:05";

}
·         The only required fields are the PF and DEFAULT fields.
·         Though it's generally a good idea to specify a VF's MAC address from the PF
·         iovctl -d <interface> -S will show all of the parameters supported by the driver
3.    iovctl -f <interface>.conf -C will create the VFs using the parameters in the .conf.
4.    To attach the newly-spawned VFs to a VM:
a.    Run "pciconf -v -l", and find the device bus/slot/function for the VF that you want to attach:
ppt0 at pci0:130:0:16:       class=0x020000 card=0x00008086 chip=0x154c8086 rev=0x01 hdr=0x00
b.    Run the bhyveload and bhyve commands as above, but add -s 4:0,passthru,130/0/16 (replace with your b/s/f from above) to the bhyve command, and add -S to both commands.
i.        The -S part is new for FreeBSD 11.0-CURRENT
5.    After the VM boots, load the PF's corresponding VF driver in the VM.
Shutting down a VM
·         Shutdown or reboot through the guest
·         Or, if that's not possible, use the bhyvectl command:
bhyvectl --force-off --vm=<vm-name>
·         To completely delete a VM, use this bhyvectl command, instead:
bhyvectl --destroy --vm=<vm-name>
Links
·         Handbook page on bhyve<https://www.freebsd.org/doc/handbook/virtualization-host-bhyve.html>
·         Wiki page on bhyve<https://wiki.freebsd.org/bhyve>
·         Note: This page has a three-line quickstart if you just want to boot up a VM with minimal customization
·         Man page for bhyve(8)<https://www.freebsd.org/cgi/man.cgi?query=bhyve&sektion=8>
·         Man page for bhyveload(8)<https://www.freebsd.org/cgi/man.cgi?query=bhyveload&sektion=8&apropos=0&manpath=FreeBSD+10.1-RELEASE>
·         UEFI & VNC graphics on bhyve<https://wiki.freebsd.org/bhyve/UEFI>

Jeff

On 5/29/18, 11:19 AM, "Sean Bruno" <owner-freebsd-net at freebsd.org on behalf of sbruno at freebsd.org> wrote:

    On 05/29/18 12:15, Kevin Bowling wrote:

    > iovctl worked at one point with ixgbe. I don’t recall if there is any

    > way to attach the vf directly if you do that but you can use ppt driver

    > to pass them into bhyve.

    >

    > On Tue, May 29, 2018 at 9:58 AM Sean Bruno <sbruno at freebsd.org

    > <mailto:sbruno at freebsd.org>> wrote:

    >

    >     Does anyone have a process for testing the VF drivers (ixgbe igb, etc)

    >     in FreeBSD without actually firing up linux to instantiate a VM or using

    >     EC2?

    >

    >     sean

    >

    I'm using iovctl with a different vendor h/w on ARM64, so I think its

    still working.

    Do you have a reference to a "how to" or whatever that someone like

    Dexter is maintaining?

    sean