interrupt latency and driver locking

[Sam Leffler]

I enabled MUTEX_PROFILING on a fast machine I've got setup as a
firewall/router.  The machine had an fxp device on one side and an em on
the other.  I then ran a bunch of netperf tests through the machine.
Unfortunately for the moment I'm stuck with 100baseT on the fxp side of the
machine so my tests were all limited by the media.  But one thing stuck out:

   max        total       count   avg name
   282          786           3   262 kern/kern_exec.c:853 (vm page queue
mute
   775         7450          33   225 kern/vfs_subr.c:3218 (mntvnode)

  1119         2789          15   185 kern/kern_fork.c:218 (Giant)

   222         2344          14   167 vm/uma_core.c:1186 (UMA lock)

   460         2972          19   156 kern/kern_exit.c:282 (vm page queue
mute
   508         2415          19   127 kern/kern_exit.c:101 (Giant)

   194         1801          16   112 kern/kern_exec.c:261 (Giant)

   332        31113         277   112 dev/fxp/if_fxp.c:1798 (fxp0)

  2435        31848         366    87 kern/kern_sysctl.c:1210 (Giant)

   901         5306          66    80 vm/vm_map.c:1256 (Giant)

    77           77           1    77 kern/uipc_syscalls.c:472 (Giant)

   182         2901          39    74 vm/vm_kern.c:328 (system map)

   507       841090       12508    67 dev/random/yarrow.c:172 (random
reseed) 
  7005        68335        1136    60 kern/kern_intr.c:533 (Giant)

    66          964          16    60 vm/vm_map.c:2241 (system map)

    88         7471         139    53 dev/em/if_em.c:1531 (em0)

   881         8032         191    42 i386/i386/trap.c:997 (Giant)

   591     10568722      271157    38 net/netisr.c:215 (netisr lock)

    38          292           8    36 vm/vm_fault.c:997 (Giant)

Note that the fxp driver holds its driver lock for an average 112 us in one
spot.  This turns out to be in fxp_tick and the likely culprit is the call
to mii_tick which is done with the lock held.  Since there's only one lock
in the driver this means interrupts are blocked during this time which is
likely to do bad things.  (The em driver comparison has a similar
bottleneck though it's <1/2 as long.)

I'm not sure if it's simple to move MII operations outside of the driver
lock but it might be worth investigating.

	Sam