Issue with epoch_drain_callbacks and unloading iavf(4) [using iflib]

[Eric Joyner]

Hi freebsd-net,

We've encountered an issue with unloading the iavf(4) driver on FreeBSD
12.1 (and stable). On a VM with two iavf(4) interfaces, if we send heavy
traffic to iavf1 and try to kldunload the driver, the kldunload process
hangs on iavf0 until iavf1 stops receiving traffic.

After some debugging, it looks like epoch_drain_callbacks() [via
if_detach_internal()] tries to switch CPUs to run on one that iavf1 is
using for RX processing, but since iavf1 is busy, it can't make the switch,
so cpu_switch() just hangs and nothing happens until iavf1's RX thread
stops being busy.

I can work around this by inserting a kern_yield(PRI_USER) somewhere in one
of the iavf txrx functions that iflib calls into (e.g.
iavf_isc_rxd_available), but that's not a proper fix. Does anyone know what
to do to prevent this from happening?

Wildly guessing, does maybe epoch_drain_callbacks() need a higher priority
than the PI_SOFT used in the group taskqueues used in iflib's RX processing?

- Eric