semaphores between processes

[Andrew Gallatin]

Daniel Eischen wrote:
> On Fri, 23 Oct 2009, Andrew Gallatin wrote:
> 
>> Daniel Eischen wrote:
>>> On Fri, 23 Oct 2009, John Baldwin wrote:
>>>
>>>> On Thursday 22 October 2009 5:17:07 pm Daniel Eischen wrote:
>>>>> On Thu, 22 Oct 2009, Andrew Gallatin wrote:
>>>>>
>>>>>> Daniel Eischen wrote:
>>>>>>> On Thu, 22 Oct 2009, Andrew Gallatin wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> We're designing some software which has to lock access to
>>>>>>>> shared memory pages between several processes, and has to
>>>>>>>> run on Linux, Solaris, and FreeBSD.  We were planning to
>>>>>>>> have the lock be a pthread_mutex_t residing in the
>>>>>>>> shared memory page.  This works well on Linux and Solaris,
>>>>>>>> but FreeBSD (at least 7-stable) does not support
>>>>>>>> PTHREAD_PROCESS_SHARED mutexes.
>>>>>>>>
>>>>>>>> We then moved on to posix semaphores.  Using sem_wait/sem_post
>>>>>>>> with the sem_t residing in a shared page seems to work on
>>>>>>>> all 3 platforms.  However, the FreeBSD (7-stable) man page
>>>>>>>> for sem_init(3) has this scary text regarding the pshared
>>>>>>>> value:
>>>>>>>>
>>>>>>>>     The sem_init() function initializes the unnamed semaphore 
>>>>>>>> pointed to
>>>>>>>> by
>>>>>>>>     sem to have the value value.  A non-zero value for pshared 
>>>>>>>> specifies
>>>> a
>>>>>>>>     shared semaphore that can be used by multiple processes, 
>>>>>>>> which this
>>>>>>>>     implementation is not capable of.
>>>>>>>>
>>>>>>>> Is this text obsolete?  Or is my test just "getting lucky"?
>>>>>>>
>>>>>>> I think you're getting lucky.
>>>>>>
>>>>>> Yes, after playing with the code some, I now see that. :(
>>>>>>
>>>>>>>> Is there recommended way to do this?
>>>>>>>
>>>>>>> I believe the only way to do this is with SYSV semaphores
>>>>>>> (semop, semget, semctl).  Unfortunately, these are not as
>>>>>>> easy to use, IMHO.
>>>>>>
>>>>>> Yes, they are pretty ugly, and we were hoping to avoid them.
>>>>>> Are there any plans to support either PTHREAD_PROCESS_SHARED
>>>>>> mutexes, or pshared posix semaphores in FreeBSD?
>>>>>
>>>>> It's planned, just not (yet) being actively worked on.
>>>>> It's a API change mostly, and then adding in all the
>>>>> compat hooks so we don't break ABI.
>>>>
>>>> There are also an alternate set of patches on threads@ to allow just 
>>>> shared
>>>> semaphores I think w/o the changes to the pthread types.  I can't 
>>>> recall
>>>> exactly what they did, but I think rrs@ was playing with using umtx 
>>>> directly
>>>> to implement some sort of process-shared primitive.
>>>
>>> That's really not the way to go.  The structs really need
>>> to become public.
>>>
>>
>> It would be great if they were, but that discussion was 6 months
>> ago, and nothing seems to have happened.  Plus we need to support
>> at least 7.X and probably 6, so any changes here might not even
>> help us.
>>
>> What is wrong  with just using umtx directly?  It seems to do
>> exactly what we need.
> 
> Because you can't do anything more than use umtx directly,
> like check for mutex types and return appropriate error
> codes.  Just look at other implementations - Solaris,
> Linux, all have their pthread_*_t as public structs.

I'm not saying that having pthread*t public, and getting all
the features of real PTHREAD_PROCESS_SHARED would not be far
better in general.  But in this case all we need is a lock around
a shared resource.  Eg, nothing fance.  So our choices seem to be
either:

1) use sysv semaphores (ick)
2) use a hand rolled spinlock (ick)
3) use some sort of hack built into our driver (ick, ick)
4) use umtx

Is there some bug or limitation in umtx that makes it inappropriate?
(beyond the obvious, like the potential to leave a resource locked
forever if the lock holder exits).

Thanks,

Drew