timeout/untimeout race conditions/crash [patch]

Alfred Perlstein alfred at freebsd.org
Fri Mar 14 20:01:41 PDT 2008


We think we tracked down a defect in timeout/untimeout in
FreeBSD.

We have reduced the problem to the following scenario:

2+ cpu system, one cpu is running softclock at the same time
another thread is running on another cpu which makes use of
timeout/untimeout.

CPU 0 is running "softclock"
CPU 1 is running "driver" with Giant held.

softclock: mtx_lock_spin(&callout_lock)
softclock: CACHES the callout structure's fields.
softclock: sees that it's a CALLOUT_LOCAL_ALLOC
softclock: executes this code:
  if (c->c_flags & CALLOUT_LOCAL_ALLOC) {
  	c->c_func = NULL;
  	c->c_flags = CALLOUT_LOCAL_ALLOC;
  	SLIST_INSERT_HEAD(&callfree, c,
  		c_links.sle);
  	curr_callout = NULL;
  } else {
  
  NOTE: that c->c_func has been set to NULL and curr_callout
        is also NULL.
softclock: mtx_unlock_spin(&callout_lock)
driver: calls untimeout(), the following sequence happens:
        mtx_lock_spin(&callout_lock);
        if (handle.callout->c_func == ftn && handle.callout->c_arg == arg)
                callout_stop(handle.callout);
        mtx_unlock_spin(&callout_lock);

  NOTE: untimeout() sees that handle.callout->c_func is not set
        to the function so it does NOT call callout_stop(9)!
driver: free's backing structure for c->c_arg.
softclock: executes callout.
softclock: likely crashes at this point due to access after free.

I have a patch I'm trying out here, but I need feedback on it.

The way the patch works is to treat CALLOUT_LOCAL_ALLOC (timeout/untimeout)
callouts the same as ~CALLOUT_LOCAL_ALLOC allocs, and moves the
freelist manipulation to the end of the callout dispatch.

Some light testing seems to have the system work.

We are doing some testing in-house to also make sure this works.

Please provide feedback.

See attached delta.

-- 
- Alfred Perlstein
-------------- next part --------------
A non-text attachment was scrubbed...
Name: kern_timeout.diff
Type: text/x-diff
Size: 1185 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20080315/a340c52a/kern_timeout.bin


More information about the freebsd-stable mailing list