How to do proper locking

[Hans Petter Selasky]

Hi,

I am looking for a safe way to access structures that can be freed. The 
solution I am looking for must not:

- hinder scaleability
- lead to use of a single lock
- lead to lock order reversal


Here is the solution I have landed on so far:


First I plan to make a reference count manager that has the following extern 
functions:

u_int32_t *ref_alloc(); // will allocate a reference count
void ref_free(u_int32_t *); // will free a reference count and increment it.
                            // Note that the pointer passed to this function
                            // is still valid after its call

u_int32_t ref_atomic_read(u_int32_t *); // will read the value 
                                        // of a reference count

void ref_atomic_increment(u_int32_t *); // will increment the value
                                        // of a reference count

Assume that we have the following structure:

struct my_struct {

  struct mtx *p_mtx;
  u_int32_t *p_ref;

  u_int8_t my_data[256];

};



At some point this structure is allocated:

static struct my_struct *ptr;

mtx_lock(lock_A);
if(ptr == NULL)
{
  ptr = malloc(...);

  if(ptr)
  {
    ptr->p_ref = ref_alloc();
    ptr->p_mtx = mtx_alloc();
  }
}
mtx_unlock(lock_A);




At some other point it is freed:

mtx_lock(lock_A);
ptr_copy = ptr;
ptr = NULL;
mtx_unlock(lock_A);

if(ptr_copy)
{
  mtx_lock(ptr_copy->p_mtx); // we want to hold this lock 
                             // to block the callback while 
                             // incrementing refcount

  ref_free(ptr_copy->p_ref); // will increment the refcount
  mtx_unlock(ptr_copy->p_mtx); 

  mtx_free(ptr_copy->p_mtx); // Note: mutex is still valid after this!
  free(ptr_copy);
}




Then at the last point we want to access this structure, but we don't want to 
hold "lock_A", but rather "ptr->p_mtx", to increase performance.

FIRST_PART:

mtx_lock(lock_A);
if(ptr)
{
  p_ref_copy = ptr->p_ref;
  ref_value = ref_atomic_read(ptr->p_ref);
  p_mtx_copy = ptr->p_mtx;
}
else
{
  p_ref_copy = NULL;
}
mtx_unlock(lock_A);

SECOND_PART:

if(p_ref_copy)
{
  mtx_lock(p_mtx_copy);

  if(ref_value == ref_atomic_read(p_ref_copy))
  {
     /* this structure is still allocated */
     CALLBACK_CODE_HERE:
  }
  else
  {
     /* this structure has been freed */
  }

  mtx_unlock(p_mtx_copy);
}


To access a memory structure safely, one needs three parameters, according to 
my theory: "p_ref_copy", "ref_value", "p_mtx_copy". Then I thought that one 
might prestore these, so that the "FIRST_PART" can be skipped, left with only 
the "SECOND_PART". Then I thought that the "SECOND_PART" could be implemented 
by existing callbacks, so that we stay out of trouble.


Any comments ?



(Hence todays computers are so fast, one might want to use a 64-bit reference 
count. So after some billion years my model will fail, but one will probably 
reboot long before that :-)


--HPS