[Bug 253473] Possible bug in iflib, observed with vmxnet3 driver on ESXI servers

[bugzilla-noreply at freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=253473

            Bug ID: 253473
           Summary: Possible bug in iflib, observed with vmxnet3 driver on
                    ESXI servers
           Product: Base System
           Version: 12.1-RELEASE
          Hardware: amd64
                OS: Any
            Status: New
          Severity: Affects Some People
          Priority: ---
         Component: kern
          Assignee: bugs at FreeBSD.org
          Reporter: shrikanth07 at gmail.com
 Attachment #222401 text/plain
         mime type:

Created attachment 222401
  --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=222401&action=edit
Boot log from Juniper VSRX box

Possible bug with iflib code observed with VMXNET3 driver. On a 12.1 RELEASE
FreeBSD VM running on ESXi server, doing a "ifconfig vmx0 media autoselect"
renders the interface unusable. You can have a ping going to the interface from
outside of VM, it stops when you do a media autoselect using ifconfig. 
My real objective was to track the way iflib processes media change so the
ifconfig command wasn't a specific need for me. The original instance of this
problem was on a Juniper VSRX box running Junos running FreeBSD stable/12
(12.1) underneath. I actually reached out to bryanv at freebsd.org on the problem
before I verified it on pristine FreeBSD stable/12 (Junos customizes FreeBSD
drivers to work with it's network stack). 

I guess when iflib layer is processing iflib_media_change() it calls
iflib_init_locked() while the driver is running and does not bother to check
the state while it attempts some of the initialization again which leaves the
interface in an inconsistent state. 

root at freebsd:~ #ifconfig vmx0 media autoselect
vmx0: device enable command failed!

To recover you have to flap the interface, ifconfig vmx0 down and then up
recovers the interface.

original observation of the issue and the email I sent to Bryan Venteicher
--------------------------------------------------------------------------

Hello Bryan,

I need to clarify a certain behavior with vmxnet3 driver and iflib on FreeBSD
stable/12.

The background on this is during vmxnet3 driver initialization probe/attach
happens fine and interface initialization is all good. A subsequent reconfigure
on the interface does the following call flow
iflib_if_init → iflib_stop → vmxnet3_stop → iflib_init_locked 
What that does underneath is a proper VMXNET3_CMD_DISABLE, VMXNET3_CMD_RESET
and eventually VMXNET3_CMD_ENABLE. No problem there.

But on a subsequent ioctl (SIOCSIFMEDIA) it does a VMXNET3_CMD_ENABLE again on
an already enabled device which fails and renders the interface in a non
working state.  The call flow seems to be 

Ifmedia_ioctl → ifm->ifm_change → iflib_media_change → vmxnet3_media_change.

vmxnet3_media_change return 0→ iflib_init_locked → vmxnet3_init
→vmxnet3_enable_device

The problem seems iflib_init_locked() does *not* care to do a iflib_stop() and
vmxnet3_media_change() being a dummy function just returns 0 indicating success
to iflib_media_change so it proceeds to call iflib_init_locked() without
calling iflib_stop(). 
Herein lies the problem and my query, iflib_init_locked ends up calling
vmxnet3_init() → vmxnet3_enable_device() which means doing VMXNET3_CMD_ENABLE a
second time, the subsequent one resulting in a failure as indicated by the log
“device enable command failed!”. (Refer the logs below captured from a boot).

The workaround to recover the interface is to flap it i.e ifconfig fxp0 down /
up to recover.

Another experiment was to do a VMXNET3_CMD_DISABLE in vmxnet3_media_change to
“support” the subsequent VMXNET3_CMD_ENABLE from iflib_init_locked. I tried
looking around for documentation/datasheet to confirm this behavior but sadly
no direct hits. Possibly look at the Linux side of things, not sure but
reaching out to you for help.

Note that the environment is Junos (Juniper OS) based on FreeBSD stable/12 for
one of our products which is an instance running on ESXI servers.

Below are the logs from the boot of a VSRX device. The ones marked with ### are
some device_printfs added to capture the call flow along with IFMEDIA_DEBUG
enabled. 

My question is should there be a sequence of VMXNET3_CMD_DISABLE /
VMXNET3_CMD_RESET before calling vmxnet3_media_change?


Vmxnet3 logs from boot (complete boot log is attached)

------------------------------------------------------

vmx0: <VMware VMXNET3 Ethernet Adapter> port 0x4000-0x400f mem
0xfd5fc000-0xfd5fcfff,0xfd5fd000-0xfd5fdfff,0xfd5fe000-0xfd5fffff irq 18 at
device 0.0 on pci3

vmx0: Using 512 TX descriptors and 512 RX descriptors
vmx0: Using 4 RX queues 4 TX queues
vmx0: failed to allocate 5 MSI-X vectors, err: 6
vmx0: Using an MSI interrupt
vmx0: LRO is disabled

Adding entry for Ethernet autoselect
ifmedia_set: target Ethernet autoselect
ifmedia_set: setting to Ethernet autoselect

Reconfiguring vmx0 to fxp0 ...
vmx0: ### iflib_if_init: called
vmx0: ### iflib_if_init_locked: called
vmx0: ### iflib_stop: called
vmx0: ### vmxnet3_stop: called!
vmx0: ### iflib_init_locked: called
vmx0: ### vmxnet3_init: called!
vmx0: ### vmxnet3_reinit_shared_data: mtu: 1514.
vmx0: ### vmxnet3_reinit_queues: called!
vmx0: ### vmxnet3_txinit: called!
vmx0: ### vmxnet3_rxinit: called!
vmx0: ### vmxnet3_enable_device: called!
vmx0: ### vmxnet3_reinit_rxfilters: called!
vmx0: ### iflib_if_ioctl: called: command: 2152753457
vmx0: ### vmxnet3_multi_set: called!
vmx0: ### iflib_if_ioctl: called: command: 3226495287
ifmedia_ioctl: switching fxp0 to Ethernet autoselect
vmx0: ### iflib_media_change: called
vmx0: ### vmxnet3_media_change: called!
vmx0: ### iflib_init_locked: called
vmx0: ### vmxnet3_init: called!
vmx0: ### vmxnet3_reinit_shared_data: mtu: 1514.
vmx0: ### vmxnet3_reinit_queues: called!
vmx0: ### vmxnet3_txinit: called!
vmx0: ### vmxnet3_rxinit: called!
vmx0: ### vmxnet3_reinit_rxfilters: called!
vmx0: device enable command failed!
vmx0: ### iflib_if_ioctl: called: command: 3227543864
vmx0: ### vmxnet3_update_admin_status: called!
vmx0: ### vmxnet3_media_status: called!
vmx0: ### iflib_if_ioctl: called: command: 2152753424
vmx0: ### iflib_if_ioctl: called: command: 2152753457
vmx0: ### vmxnet3_multi_set: called!

-- 
You are receiving this mail because:
You are the assignee for the bug.