Transitioning if_addr_lock to an rwlock

[John Baldwin]

On Friday, December 23, 2011 12:13:44 pm Arnaud Lacombe wrote:
> Hi,
> 
> On Thu, Dec 22, 2011 at 11:30 AM, John Baldwin <jhb at freebsd.org> wrote:
> > Another bit of lock contention I ran into between a device driver doing slow
> > MAC filter updates and the receive path is IF_ADDR_LOCK().  NIC device drivers
> > typically hold this lock while iterating the list of multicast addresses to
> > program their MAC filter.  OTOH, ip_input() uses this lock to check to see if
> > an incoming packet is a broadcast packet or not.  So even with the pcbinfo
> > contention from my previous patch addressed, I still ran into a problem with
> > IF_ADDR_LOCK().  We already have some partial support for making this lock be
> > an rwlock (the APIs that drivers now use implies an rlock), so I finished the
> > transition and checked various non-driver users to see which ones could use a
> > read lock (most uses can).  The current patch I have for this is on 8, but if
> > folks think this is a good idea I can work on porting it to HEAD.  For HEAD
> > the strategy I would use would be to split this up into 2 phases:
> >
> > 1) Add IF_ADDR locking macros to differentiate read locks vs write locks along
> >   with appropriate assertion macros.  Update current users of the locking
> >   macros to use either read or write locks explicitly.  To preserve KPI,
> >   the existing LOCK/UNLOCK macros would map to write lock operations.  In
> >   the initial patch, the locking would still be implemented via a mtx.
> >
> > 2) Replace the mutex with an rwlock and update the locking macros as
> >   appropriate.
> >
> out of curiosity, what do you expect from the conversion ? performance
> improvement ? latency improvement ?
> 
> Does this particular lock show up in any significant way in lock
> profiling that make the change noticeable ?

More of a latency / don't drop packets under load improvement.  Specifically,
I have a latency-sensitive workload where packets are being dropped on the
card because the ISR is contending on this lock with another thread on another
CPU that is joining a multicast group on a socket causing the MAC filters on
the same NIC to be updated.  (The packets are dropped because the ISR can't
drain the packets fast enough while it is stuck on the lock, this causes the
receive ring to fill up and eventually the card runs out of room and drops
incoming traffic.)

-- 
John Baldwin