kern/57722: uidinfo list corruption
Ed Maste
emaste at sandvine.com
Tue Oct 7 15:20:10 PDT 2003
>Number: 57722
>Category: kern
>Synopsis: uidinfo list corruption
>Confidential: no
>Severity: serious
>Priority: medium
>Responsible: freebsd-bugs
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Tue Oct 07 15:20:08 PDT 2003
>Closed-Date:
>Last-Modified:
>Originator: Ed Maste
>Release: FreeBSD 4.70.0040. i386
>Organization:
Sandvine Incorporated
>Environment:
System: FreeBSD TPC-D3-28 4.70.0040. FreeBSD 4.70.0040. #4:
20031007_15:42:17 i386
>Description:
(Sorry if this ends up as a duplicate PR; it seems FreeBSD.org's mx didn't
like our slightly broken mail setup and my original PR has now bounced
between two postfix MTAs and is currently stuck in an Exchange mail queue
with no way to recover it. You don't want to know.)
While running a test case here that forks off clients and servers which
call setuid then establish tcp connections we reliably get a panic on an
INVARIANTS kernel. It appears to be similar to the crfree() race condition
that I recently looked at.
In the case that I observed it appears that the setuid() syscall causes
uifind() to look in the uihash and walk the list looking for the appropriate
uidinfo record or creating a new one if necessary. This is interrupted by
the stack which ends up calling uifree(), corrupting the record uifind()
was examining.
In order to catch it I set a global variable in uifind() that gets checked
in uifree(). Uifree() panics if the flag is set. That resulted in the
following stack trace:
#0 dumpsys () at /d2/emaste/src/sys/kern/kern_shutdown.c:492
#1 0xc01add23 in boot (howto=260)
at /d2/emaste/src/sys/kern/kern_shutdown.c:321
#2 0xc01ae1c5 in panic (fmt=0xc02e6fbf "uifree called during uifind")
at /d2/emaste/src/sys/kern/kern_shutdown.c:607
#3 0xc01ad6ba in uifree (uip=0xc3423760)
at /d2/emaste/src/sys/kern/kern_resource.c:742
#4 0xc01ab9f3 in crfree (cr=0xc369de80)
at /d2/emaste/src/sys/kern/kern_prot.c:1023
#5 0xc01cc7bb in sodealloc (so=0xde0fdfc0)
at /d2/emaste/src/sys/kern/uipc_socket.c:210
#6 0xc01cc8bb in sofree (so=0xde0fdfc0)
at /d2/emaste/src/sys/kern/uipc_socket.c:265
#7 0xc01f87b1 in in_pcbdetach (inp=0xde5e4640)
at /d2/emaste/src/sys/netinet/in_pcb.c:578
#8 0xc0208a9d in tcp_close (tp=0xde5e4700)
at /d2/emaste/src/sys/netinet/tcp_subr.c:754
#9 0xc020ab1a in tcp_timer_2msl (xtp=0xde5e4700)
at /d2/emaste/src/sys/netinet/tcp_timer.c:219
#10 0xc01b3f19 in softclock () at /d2/emaste/src/sys/kern/kern_timeout.c:131
#11 0xc02a5c14 in splz_swi ()
#12 0xc0254984 in interlocked_sleep (lk=0xc033367c, op=1, ident=0xdf327b84,
flags=17, wmesg=0xc02fa41f "drainvp", timo=0)
at /d2/emaste/src/sys/ufs/ffs/ffs_softdep.c:329
#13 0xc025a236 in drain_output (vp=0xdf327b40, islocked=1)
at /d2/emaste/src/sys/ufs/ffs/ffs_softdep.c:4913
#14 0xc0259012 in softdep_fsync_mountdev (vp=0xdf327b40)
at /d2/emaste/src/sys/ufs/ffs/ffs_softdep.c:4056
#15 0xc025d35e in ffs_fsync (ap=0xe0758cf4)
at /d2/emaste/src/sys/ufs/ffs/ffs_vnops.c:134
#16 0xc025bfd2 in ffs_sync (mp=0xc35ce200, waitfor=2, cred=0xc2053700,
p=0xc035f9c0) at vnode_if.h:558
#17 0xc01df577 in sync (p=0xc035f9c0, uap=0x0)
at /d2/emaste/src/sys/kern/vfs_syscalls.c:577
#18 0xc01adabe in boot (howto=256)
at /d2/emaste/src/sys/kern/kern_shutdown.c:240
#19 0xc01ae1c5 in panic (fmt=0xc02e6fbf "uifree called during uifind")
at /d2/emaste/src/sys/kern/kern_shutdown.c:607
#20 0xc01ad6ba in uifree (uip=0xc3423900)
at /d2/emaste/src/sys/kern/kern_resource.c:742
#21 0xc01ab9f3 in crfree (cr=0xc36a3e80)
at /d2/emaste/src/sys/kern/kern_prot.c:1023
#22 0xc01cc7bb in sodealloc (so=0xde0fe380)
at /d2/emaste/src/sys/kern/uipc_socket.c:210
#23 0xc01cc8bb in sofree (so=0xde0fe380)
at /d2/emaste/src/sys/kern/uipc_socket.c:265
#24 0xc01f87b1 in in_pcbdetach (inp=0xde5e4a80)
at /d2/emaste/src/sys/netinet/in_pcb.c:578
#25 0xc0208a9d in tcp_close (tp=0xde5e4b40)
at /d2/emaste/src/sys/netinet/tcp_subr.c:754
#26 0xc020ab1a in tcp_timer_2msl (xtp=0xde5e4b40)
at /d2/emaste/src/sys/netinet/tcp_timer.c:219
#27 0xc01b3f19 in softclock () at /d2/emaste/src/sys/kern/kern_timeout.c:131
#28 0xc02a5b6b in doreti_swi ()
#29 0xc01ad5c7 in uicreate (uid=2853)
at /d2/emaste/src/sys/kern/kern_resource.c:697
#30 0xc01ad690 in uifind (uid=2853)
at /d2/emaste/src/sys/kern/kern_resource.c:730
#31 0xc01abbe3 in change_ruid (p=0xdc352d80, ruid=2853)
at /d2/emaste/src/sys/kern/kern_prot.c:1183
#32 0xc01ab22a in setuid (p=0xdc352d80, uap=0xe0758f80)
at /d2/emaste/src/sys/kern/kern_prot.c:433
#33 0xc02b5985 in syscall2 (frame={tf_fs = 47, tf_es = 47, tf_ds = 47,
tf_edi = -1077936948, tf_esi = -1077937120, tf_ebp = -1077937288,
tf_isp = -529166380, tf_ebx = 4, tf_edx = 1, tf_ecx = 134520696,
tf_eax = 23, tf_trapno = 12, tf_err = 2, tf_eip = 1745479444,
tf_cs = 31, tf_eflags = 643, tf_esp = -1077937412, tf_ss = 47})
at /d2/emaste/src/sys/i386/i386/trap.c:1193
#34 0xc02a1abb in Xint0x80_syscall ()
Note that crfree (#21) is wrapped in splhigh from the cred fix.
>How-To-Repeat:
Make a lot of calls to setuid() while TCP sockets are connecting and timing
out.
>Fix:
The patch below fixes the problem here. Splhigh() might not be needed
in uifree(), since the cases I was looking into all come from the stack
calling crfree(), which is now wrapped in splhigh().
uifind() or uifree() is also called from kern_vfs, and I have no idea if
any of these could cause the same problem (by interrupting a syscall).
If they couldn't, I'd think splnet() would be sufficient.
Index: kern_resource.c
===================================================================
RCS file: /cvs/src/sys/kern/kern_resource.c,v
retrieving revision 1.55.2.5
retrieving revision 1.55.2.5.1000.1
diff -p -c -3 -r1.55.2.5 -r1.55.2.5.1000.1
*** kern_resource.c 3 Nov 2001 01:41:08 -0000 1.55.2.5
--- kern_resource.c 6 Oct 2003 21:10:21 -0000 1.55.2.5.1000.1
*************** uifind(uid)
*** 717,727 ****
--- 717,729 ----
uid_t uid;
{
struct uidinfo *uip;
+ int s = splhigh();
uip = uilookup(uid);
if (uip == NULL)
uip = uicreate(uid);
uip->ui_ref++;
+ splx(s);
return (uip);
}
*************** uifree(uip)
*** 730,735 ****
--- 732,738 ----
struct uidinfo *uip;
{
+ int s = splhigh();
if (--uip->ui_ref == 0) {
if (uip->ui_sbsize != 0)
/* XXX no %qd in kernel. Truncate. */
*************** uifree(uip)
*** 740,747 ****
--- 743,752 ----
uip->ui_uid, uip->ui_proccnt);
LIST_REMOVE(uip, ui_hash);
FREE(uip, M_UIDINFO);
+ splx(s);
return (1);
}
+ splx(s);
return (0);
}
>Release-Note:
>Audit-Trail:
>Unformatted:
More information about the freebsd-bugs
mailing list