Issue with epoch_drain_callbacks and unloading iavf(4) [using iflib]

[Mark Johnston]

On Wed, Mar 11, 2020 at 04:32:40PM -0700, Eric Joyner wrote:
> Mark,
> 
> I did get some time to get back and retry this; however your second patch
> still doesn't solve the problem. Looking into it a bit, it looks like the
> kldunload process isn't hitting the code you've changed; it's hanging in
> epoch_wait_preempt() in if_detach_internal(), which is immediately before
> epoch_drain_callbacks().
> 
> I did a kernel dump while it was hanging, and this is the backtrace for the
> kldunload process:

I see.  I think the callback can be made much simpler and avoid the
problematic sched_bind() calls.  I wrote a patch that allows waiting
threads to lend scheduling priority to a preempted thread blocked in an
epoch section, based on some code I wrote to implement preemptible SMR
sections.  If waiting for a running thread, the callback just spins.

This might be enough to solve your problem, I posted the two lightly
tested patches here:
https://reviews.freebsd.org/D24214
https://reviews.freebsd.org/D24215

If we hit a situation where a reader is preempted and then its CPU is
hogged by a high-priority kernel thread, this still won't be enough, but
I suspect it'll solve your case.  Would you be able to test?