Dynamic VF mac address manipulation / Bhyve

Hi -

I am slowly converting a Linux based qemu/kvm solution to FreeBSD
Bhyve where VMs have mac addresses assigned to them.


Linux allows individual VF entries per $iface to be manipulated:

ip link set ${iface} vf $vface mac $vfmac

The VF is then passed into qemu/kvm and the VM sees the mac address
as assigned above ($vfmac):

-device vfio-pci,host=81:02.1,bus=pci.1,id=vfnet0



iovctl allows for mac address control of all VFs under an interface
at one time - and requires the destruction and re-creation of all VF
entries in order to modify a single entry - which breaks existing
running VMs.

Is there a way I have missed to modify the mac address of a single
VF which is presented to a VM?


I have tried the following steps.

# iovctl -C -f /root/iov-cc0.conf   # Create, for instance, 4 VFs agains cc0.

# pciconf -l | grep vf
t6vf0@pci0:65:0:8:	class=0x020000 rev=0x00 hdr=0x00 vendor=0x1425 device=0x6808 subvendor=0x1425 subdevice=0x0000
t6vf1@pci0:65:0:12:	class=0x020000 rev=0x00 hdr=0x00 vendor=0x1425 device=0x6808 subvendor=0x1425 subdevice=0x0000
t6vf2@pci0:65:0:16:	class=0x020000 rev=0x00 hdr=0x00 vendor=0x1425 device=0x6808 subvendor=0x1425 subdevice=0x0000
t6vf3@pci0:65:0:20:	class=0x020000 rev=0x00 hdr=0x00 vendor=0x1425 device=0x6808 subvendor=0x1425 subdevice=0x0000

ccv0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=66ec07bb<RXCSUM,TXCSUM,VLAN_MTU,....
	ether 02:00:00:00:00:00
	media: Ethernet 100GBase-SR4 <full-duplex> (100GBase-SR4 <full-duplex,rxpause,txpause>)
	status: active
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ccv1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=66ec07bb<RXCSUM,TXCSUM,VLAN_MTU,....
	ether 02:00:00:00:00:00
	media: Ethernet 100GBase-SR4 <full-duplex> (100GBase-SR4 <full-duplex,rxpause,txpause>)
	status: active
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ccv2: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=66ec07bb<RXCSUM,TXCSUM,VLAN_MTU,....
	ether 02:00:00:00:00:00
	media: Ethernet 100GBase-SR4 <full-duplex> (100GBase-SR4 <full-duplex,rxpause,txpause>)
	status: active
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ccv3: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=66ec07bb<RXCSUM,TXCSUM,VLAN_MTU,....
	ether 02:00:00:00:00:00
	media: Ethernet 100GBase-SR4 <full-duplex> (100GBase-SR4 <full-duplex,rxpause,txpause>)
	status: active
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>


Note: initial mac 02:00:00:00:00:00 indicating not-in-use and/or available.

# ifconfig ccv3 ether 02:22:33:44:55:66

# ifconfig ccv3
ccv3: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=66ec07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWRXTSTMP,NOMAP,VXLAN_HWCSUM,VXLAN_HWTSO>
	ether 02:22:33:44:55:66
	hwaddr 02:00:00:00:00:00
	media: Ethernet 100GBase-SR4 <full-duplex> (100GBase-SR4 <full-duplex,rxpause,txpause>)
	status: active
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

Ether now shows with new mac - hwaddr the original as assigned by iovctl.

Now re-assign the VF device to the pass-thru driver:

devctl set driver -f pci0:65:0:20 ppt

The VF device can now be passed into bhyve (65/0/20) but the new
value 02:22:33:44:55:66 is lost and the prior value 02:00:00:00:00:00
is seen by the vm - but we want 02:22:33:44:55:66. So the new mac is
a property of the driver, not the VF.


At this point I'm wondering if adding a -U (update) option to iovctl
or a new standalone program to target an individual iface/vf pair
for update might be the answer.


Comments welcome.

Thanks,
John