[Bug 289569] Issues with SR-IOV functionality

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=289569

            Bug ID: 289569
           Summary: Issues with SR-IOV functionality
           Product: Base System
           Version: 14.3-RELEASE
          Hardware: Any
                OS: Any
            Status: New
          Severity: Affects Some People
          Priority: ---
         Component: bin
          Assignee: bugs@FreeBSD.org
          Reporter: cryintothebluesky@gmail.com

I tried to configure SR-IOV interfaces with a network card based on Intel X520
82599ES and noticed a number of issues:

1. The rc.d scripts execute iovctl in the wrong order.

It seems this network interface card must be brought up before virtual
interfaces can be configured, otherwise executing iovctl results in the
following errors:

ixv0: <Intel(R) X520 82599 Virtual Function> at device 0.128 on pci3
ixv0: ...reset_hw() failed with error -43
ixv0: IFDI_ATTACH_PRE failed 5
device_attach: ixv0 attach returned 5

However, adding the following lines to /etc/rc.conf does not help

ifconfig_ix0="up mtu 9000"
ifconfig_ix1="up mtu 9000"
iovctl_files="/etc/sr-iov/ix0.conf /etc/sr-iov/ix1.conf"

because /etc/rc.d/netif REQUIREs /etc/rc.d/iovctl, hence iovctl always runs
first and it always fails to configure virtual interfaces as the physical
interfaces are not yet up.

2. The parser for iovctl.conf seems a bit primitive

If you create the following config file, where the opening brace is on a new
line:

PF 
{
  device : ix0;
  num_vfs: 1;
}

then you get a cryptic and unhelpful error message:

# iovctl -C -f ix0.conf 
iovctl: Could not open 'ix0.conf' for reading: No error: 0

The parser should cope with either

PF {
}

or

PF
{
}

or at the very least provide a better error message stating the exact syntax
error.

3. There seems to be no way to configure multiple TX and RX queues for virtual
interfaces.

The following settings in /boot/loader.conf get completely ignored:

dev.ixv.0.iflib.override_qs_enable=1
dev.ixv.0.iflib.override_nrxqs=4
dev.ixv.0.iflib.override_ntxqs=4

and all virtual interfaces end up with 1 TX and 1 RX queue:

ixv0: <Intel(R) X520 82599 Virtual Function> at device 0.128 on pci3
ixv0: Using 2048 TX descriptors and 2048 RX descriptors
ixv0: Using 1 RX queues 1 TX queues

The "Intel® 82599 SR-IOV Driver Companion Guide" on page 12 states that for
each virtual interface, resources are allocated via pools, where each pool can
be configured with different number of queues:

Number of Pools | Queue Pairs per Pool | Total Queues per Pool
----------------+----------------------+----------------------
      16        |          8           |          16 
----------------+----------------------+----------------------
      32        |          4           |          8 
----------------+----------------------+----------------------
      64        |          2           |          4

So if I only need a maximum of 16 virtual interfaces, I could configure 16
queues (8 TX + 8 RX) per each virtual interface and get better scalability per
VM on SMP hardware. However this configuration does not seem to be available
anywhere on FreeBSD.

4. Changing default MTU on virtual interfaces breaks network connectivity.

If I change MTU on ixv* interfaces from 1500 to 9000, then this breaks network
connectivity. If larger MTUs are not supported by the driver or hardware, this
needs to be documented and ideally running ifconfig with larger MTU should
result in error message. At the moment ix(4) man page explicitly states that
MTUs of up to 9710 bytes are supported, so the assumption here is that it also
applies to the SR-IOV virtual interfaces.

-- 
You are receiving this mail because:
You are the assignee for the bug.