kern/111260: FreeBSD kernel dead lock and a solution

[Zhouyi Zhou]

>Number:         111260
>Category:       kern
>Synopsis:       FreeBSD kernel dead lock and a solution
>Confidential:   no
>Severity:       critical
>Priority:       high
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Thu Apr 05 09:20:02 GMT 2007
>Closed-Date:
>Last-Modified:
>Originator:     Zhouyi Zhou
>Release:        FreeBSD 5-7
>Organization:
Institute of Software, Chinese Academy of Sciences
>Environment:
FreeBSD zzy 6.0-RELEASE FreeBSD 6.0-RELEASE #5 i386
>Description:
In the recently testing of FreeBSD using people.freebsd.org/~pho/stress/src/stress2.tgz, when running the 7 tests simulatanouly, the FreeBSD kernel will goto dead lock after three or more days .

The reason is as follows:
In function vm_fault at vm/vm_fault.c

299         fs.vp = vnode_pager_lock(fs.first_object);
300         KASSERT(fs.vp == NULL || !fs.map->system_map,

The kernel will try to get a lock on fs.vp with the fs.map still locks.

while in function do_execve at kern/kern_exec.c

462         if (p->p_sysent->sv_copyout_strings)
463                 stack_base = (*p->p_sysent->sv_copyout_strings)(imgp);
464         else
465                 stack_base = exec_copyout_strings(imgp);
466 
467         /*
468          * If custom stack fixup routine present for this process
469          * let it do the stack setup.
470          * Else stuff argument count as first item on stack
471          */
472         if (p->p_sysent->sv_fixup != NULL)
473                 (*p->p_sysent->sv_fixup)(&stack_base, imgp);
474         else
475                 suword(--stack_base, imgp->args->argc);
The copyout function may cause vm_fault on line 463 or 465, which will try to lock the kernel_map->root->object.sub_map, which is locked by another process in vm_fault above. While at mean time the imgp->vp is still got locked.      

>How-To-Repeat:
using people.freebsd.org/~pho/stress/src/stress2.tgz, when running the 7 tests simulatanouly, the FreeBSD kernel will goto dead lock after three or more days .

>Fix:
Add  VOP_UNLOCK(imgp->vp, 0, td)
before 
462         if (p->p_sysent->sv_copyout_strings)
463                 stack_base = (*p->p_sysent->sv_copyout_strings)(imgp);
464         else
And add  vn_lock(imgp->vp, LK_EXCLUSIVE | LK_RETRY, td);
after
474         else
475                 suword(--stack_base, imgp->args->argc);
>Release-Note:
>Audit-Trail:
>Unformatted: