interrupt latency and driver locking
jdp at polstra.com
Sat Sep 20 09:48:27 PDT 2003
On 19-Sep-2003 Sam Leffler wrote:
> I enabled MUTEX_PROFILING on a fast machine I've got setup as a
> firewall/router. The machine had an fxp device on one side and an em on
> the other. I then ran a bunch of netperf tests through the machine.
> Unfortunately for the moment I'm stuck with 100baseT on the fxp side of the
> machine so my tests were all limited by the media. But one thing stuck out:
> Note that the fxp driver holds its driver lock for an average 112 us in one
> spot. This turns out to be in fxp_tick and the likely culprit is the call
> to mii_tick which is done with the lock held.
Agreed. The MII calls usually have lots of DELAY() calls in them.
> I'm not sure if it's simple to move MII operations outside of the driver
> lock but it might be worth investigating.
It seems like it ought to be possible to at least lock at a finer
grain for those calls. The driver lock is held because the MII
routines call back into the driver and use it to do the bit-twiddling
necessary to shift data into and out of the PHY registers via the
PHY's serial interface. That's normally pretty independent of
anything else that's going on in the chip, so it could probably be
done under a separate lock.
Another idea would be to substitute something else for DELAY() calls
of more than a few microseconds. I am not very up-to-date on -current
these days, but as far as I know DELAY() is still a spin-wait. If
the delay is for more than a few microseconds it would probably be
better to switch to a different runnable thread and come back later.
In fact, CPU speeds may have reached the point where that is always
the right thing to do.
Finally, the entire mii_tick() nonsense could be thrown out for many
modern network adapters. Both the Broadcom and Intel gigabit chips
are capable of interrupting on interesting link events, so there is no
need at all to poll the PHY every second. That doesn't appear to be
true of the fxp devices, though. They need to be polled. Still, you
can avoid the expensive PHY reads and writes most of the time. For
example, if you have received any packets on the interface in the past
second, you can assume that link is good and never touch the PHY at
More information about the freebsd-arch