[Bug 265997] compat10 semaphore interface internal race may lead to application hang

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=265997

            Bug ID: 265997
           Summary: compat10 semaphore interface internal race may lead to
                    application hang
           Product: Base System
           Version: CURRENT
          Hardware: Any
                OS: Any
            Status: New
          Severity: Affects Only Me
          Priority: ---
         Component: kern
          Assignee: bugs@FreeBSD.org
          Reporter: firk@cantconnect.ru

The bug was introduced in r349951 r350478 which was also MFCed to 12.x as
r351789

Previously, any unexpected cases will lead to just exiting syscall, leaving
the decision to the userspace sem wrapper (which was done that correctly).

After these patches, non-zero sem->_has_waiters will lead to going straightly
to umtxq_sleep() without checking if sem->_count is non-zero. Non-zero
sem->_has_waiters may happen spuriously due to the same code: it doesn't
clear this flag after cancelling sleep due to non-zero sem->_count.


1. make sem_wait() and sem_post() race:
- sem_wait() check _count in userspace, sees it is zero, calls kernel ->
do_sem_wait()
- sem_post() sets _count to 1 and calls do_sem_wake() which does nothing
because waiter-thread still not in queue
- do_sem_wait() sets _has_waiters=1, then sees _count==1 and exits
- userspace sem_wait() decrements _count back to 0 _has_waiters left non-zero

2. make them race again:
- sem_wait() check _count in userspace, sees it is zero, calls kernel ->
do_sem_wait()
- sem_post() sets _count to 1 again, not touching _has_waiters, and calls
do_sem_wake() which does nothing again, because waiter-thread not in queue
- do_sem_wait() fails to set _has_waiters 0->1 because it already non-zero, and
then goes to umtxq_sleep()
- umtxq_sleep() has no chance to wake up without another sem_post(), despite
the fact that _count==1

-- 
You are receiving this mail because:
You are the assignee for the bug.