kern/111260: FreeBSD kernel dead lock and a solution

[Kris Kennaway]

The following reply was made to PR kern/111260; it has been noted by GNATS.

From: Kris Kennaway <kris at obsecurity.org>
To: Zhouyi Zhou <zhouyi04 at ios.cn>
Cc: freebsd-gnats-submit at FreeBSD.org
Subject: Re: kern/111260: FreeBSD kernel dead lock and a solution
Date: Fri, 6 Apr 2007 23:25:04 -0400

 On Thu, Apr 05, 2007 at 09:04:57AM +0000, Zhouyi Zhou wrote:
 
 > In the recently testing of FreeBSD using people.freebsd.org/~pho/stress/src/stress2.tgz, when running the 7 tests simulatanouly, the FreeBSD kernel will goto dead lock after three or more days .
 > 
 > The reason is as follows:
 > In function vm_fault at vm/vm_fault.c
 > 
 > 299         fs.vp = vnode_pager_lock(fs.first_object);
 > 300         KASSERT(fs.vp == NULL || !fs.map->system_map,
 > 
 > The kernel will try to get a lock on fs.vp with the fs.map still locks.
 > 
 > while in function do_execve at kern/kern_exec.c
 > 
 > 462         if (p->p_sysent->sv_copyout_strings)
 > 463                 stack_base = (*p->p_sysent->sv_copyout_strings)(imgp);
 > 464         else
 > 465                 stack_base = exec_copyout_strings(imgp);
 > 466 
 > 467         /*
 > 468          * If custom stack fixup routine present for this process
 > 469          * let it do the stack setup.
 > 470          * Else stuff argument count as first item on stack
 > 471          */
 > 472         if (p->p_sysent->sv_fixup != NULL)
 > 473                 (*p->p_sysent->sv_fixup)(&stack_base, imgp);
 > 474         else
 > 475                 suword(--stack_base, imgp->args->argc);
 > The copyout function may cause vm_fault on line 463 or 465, which will try to lock the kernel_map->root->object.sub_map, which is locked by another process in vm_fault above. While at mean time the imgp->vp is still got locked.      
 
 Can you please provide backtraces that lead you to this conclusion?
 
 Thanks,
 Kris