From dixit at netapp.com Sun Oct 5 03:44:56 2008 From: dixit at netapp.com (Amol Dixit) Date: Sun Oct 5 03:45:02 2008 Subject: SIGTRAP during thr_new syscall In-Reply-To: <48DB0950.3060405@freebsd.org> References: <48DAABAF.9030709@netapp.com> <48DB0950.3060405@freebsd.org> Message-ID: <48E83836.6090305@netapp.com> David, I tried your patch, but it doesn't seem to fix the problem because even though _pthread_create() calls thr_new(), this system call is never made for the first thread created. Breakpoint set in kernel code of thr_new is hit only for the initial_thread (_libpthread_init())...skips the first thread...and then is hit again for 2nd thread onwards. This patch may be necessary, but it doesn't fix my issue of SIGTRAP on thread_start(). Any clue why the first thread created using pthread_create() doesn't enter thr_new() syscall inside the kernel at all? I get the same SIGTRAP if I single-step through this thr_new() line in _pthread_create() using _local_ gdb. I am running freebsd6.0 configured for SYSTEM_SCOPE threading (1:1). Thanks, Amol David Xu wrote: > Amol Dixit wrote: >> Hi, >> I am seeing an unexpected SIGTRAP being reported to gdbserver when >> the debugged process creates a new thread via the _pthread_create() >> call of libthr library. [libthr/thread/thr_create.c,v 1.22.4.1, >> Freebsd 6.0] >> Gdbserver has internally set a breakpoint on address of >> _thread_bp_create() so that it gets notified on thread creation and >> is expecting a SIGTRAP at address (stop pc) of _thread_bp_create(). >> But instead SIGTRAP happens as a side-effect of thr_new() system call >> and the stop pc at that point is that of routine thread_start() which >> is the starting function of new thread. So gdbserver cannot match >> expected breakpoint (ie. _thread_bp_create) and is confused. >> For testing purpose, if I call _thread_bp_create() before thr_new() >> in _pthread_create(), I get the _expected_ SIGTRAP with address of >> _thread_bp_create. But that is not the fix. >> Does anyone have any idea about this SIGTRAP being reported to >> tracing process gdbserver as part of thr_new? Where is it originating >> from and why? >> Thanks, >> Amol >> > > I found kernel clears trap flag for new process but not for new thread > in cpu_fork(), you may try following patch: > > Index: i386/i386/vm_machdep.c > =================================================================== > --- i386/i386/vm_machdep.c (revision 183337) > +++ i386/i386/vm_machdep.c (working copy) > @@ -413,6 +413,15 @@ > bcopy(td0->td_frame, td->td_frame, sizeof(struct trapframe)); > > /* > + * If the current thread has the trap bit set (i.e. a debugger had > + * single stepped the process to the system call), we need to clear > + * the trap flag from the new frame. Otherwise, the new thread will > + * receive a (likely unexpected) SIGTRAP when it executes the first > + * instruction after returning to userland. > + */ > + td->td_frame->tf_eflags &= ~PSL_T; > + > + /* > * Set registers for trampoline to user mode. Leave space for the > * return address on stack. These are the kernel mode register > values. > */ From bugmaster at FreeBSD.org Mon Oct 6 11:07:03 2008 From: bugmaster at FreeBSD.org (FreeBSD bugmaster) Date: Mon Oct 6 11:09:11 2008 Subject: Current problem reports assigned to freebsd-threads@FreeBSD.org Message-ID: <200810061107.m96B72cU035639@freefall.freebsd.org> Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o threa/127225 threads bug in lib/libthr/thread/thr_init.c o threa/126950 threads [patch] rtld(1): rtld malloc is thread-unsafe o kern/126128 threads [patch] pthread_condattr_getpshared is broken o threa/122923 threads 'nice' does not prevent background process from steali o threa/121336 threads lang/neko threading ok on UP, broken on SMP (FreeBSD 7 o threa/118715 threads kse problem o threa/116668 threads can no longer use jdk15 with libthr on -stable SMP o threa/116181 threads /dev/io-related io access permissions are not propagat o threa/115211 threads pthread_atfork misbehaves in initial thread o threa/110636 threads [request] gdb(1): using gdb with multi thread applicat o threa/110306 threads apache 2.0 segmentation violation when calling gethost o threa/103975 threads Implicit loading/unloading of libpthread.so may crash o threa/101323 threads fork(2) in threaded programs broken. s threa/100815 threads FBSD 5.5 broke nanosleep in libc_r s threa/94467 threads send(), sendto() and sendmsg() are not correct in libc s threa/84483 threads problems with devel/nspr and -lc_r on 4.x o threa/83914 threads [libc] popen() doesn't work in static threaded program o threa/80992 threads abort() sometimes not caught by gdb depending on threa o threa/80435 threads panic on high loads o threa/79887 threads [patch] freopen() isn't thread-safe o threa/79683 threads svctcp_create() fails if multiple threads call at the s threa/76694 threads fork cause hang in dup()/close() function in child (-l s threa/76690 threads fork hang in child for -lc_r o threa/75374 threads pthread_kill() ignores SA_SIGINFO flag o threa/75273 threads FBSD 5.3 libpthread (KSE) bug o threa/72953 threads fork() unblocks blocked signals w/o PTHREAD_SCOPE_SYST o threa/70975 threads [sysvipc] unexpected and unreliable behaviour when usi s threa/69020 threads pthreads library leaks _gc_mutex s threa/49087 threads Signals lost in programs linked with libc_r s threa/48856 threads Setting SIGCHLD to SIG_IGN still leaves zombies under s threa/40671 threads pthread_cancel doesn't remove thread from condition qu s threa/39922 threads [threads] [patch] Threaded applications executed with s threa/37676 threads libc_r: msgsnd(), msgrcv(), pread(), pwrite() need wra s threa/34536 threads accept() blocks other threads s kern/32295 threads [libc_r] [patch] pthread(3) dont dequeue signals s threa/30464 threads pthread mutex attributes -- pshared s threa/24632 threads libc_r delicate deviation from libc in handling SIGCHL s threa/24472 threads libc_r does not honor SO_SNDTIMEO/SO_RCVTIMEO socket o 38 problems total. From petri at helenius.fi Mon Oct 6 21:29:10 2008 From: petri at helenius.fi (Petri Helenius) Date: Mon Oct 6 21:29:16 2008 Subject: getrusage with pthreads Message-ID: <48EA8320.5050500@helenius.fi> Is there a way to get thread-specific CPU utilization like the getrusage returns for the whole process? Looking at the kernel sources the utilization is accumulated on thread basis and then summed up but it would be useful to be able to determine utilization on a single thread between two checkpoints. Would pmc or ITIMER_PROF help? Pete From davidxu at freebsd.org Tue Oct 7 03:30:46 2008 From: davidxu at freebsd.org (David Xu) Date: Tue Oct 7 03:30:52 2008 Subject: getrusage with pthreads In-Reply-To: <48EA8320.5050500@helenius.fi> References: <48EA8320.5050500@helenius.fi> Message-ID: <48EAD851.7000404@freebsd.org> Petri Helenius wrote: > > Is there a way to get thread-specific CPU utilization like the getrusage > returns for the whole process? Looking at the kernel sources the > utilization is accumulated on thread basis and then summed up but it > would be useful to be able to determine utilization on a single thread > between two checkpoints. Would pmc or ITIMER_PROF help? > > Pete > You can use clock_gettime(CLOCK_THREAD_CPUTIME_ID), but I admit getrusage like function is better. Regards, David Xu From petri at helenius.fi Tue Oct 7 05:02:52 2008 From: petri at helenius.fi (Petri Helenius) Date: Tue Oct 7 05:03:01 2008 Subject: getrusage with pthreads In-Reply-To: <48EAD851.7000404@freebsd.org> References: <48EA8320.5050500@helenius.fi> <48EAD851.7000404@freebsd.org> Message-ID: <48EAED75.9010900@helenius.fi> David Xu wrote: > > You can use clock_gettime(CLOCK_THREAD_CPUTIME_ID), but I admit > getrusage like function is better. > Is this implemented in FreeBSD starting from some specific version? I cannot seem to find the header. Pete > > From davidxu at freebsd.org Tue Oct 7 05:09:47 2008 From: davidxu at freebsd.org (David Xu) Date: Tue Oct 7 05:09:53 2008 Subject: getrusage with pthreads In-Reply-To: <48EAED75.9010900@helenius.fi> References: <48EA8320.5050500@helenius.fi> <48EAD851.7000404@freebsd.org> <48EAED75.9010900@helenius.fi> Message-ID: <48EAEF87.8050803@freebsd.org> Petri Helenius wrote: > David Xu wrote: >> >> You can use clock_gettime(CLOCK_THREAD_CPUTIME_ID), but I admit >> getrusage like function is better. >> > Is this implemented in FreeBSD starting from some specific version? I > cannot seem to find the header. > > Pete > It only exists in -CURRENT. From qj at huawei.com Tue Oct 7 11:54:11 2008 From: qj at huawei.com (=?gb2312?B?x/G9ow==?=) Date: Tue Oct 7 11:54:30 2008 Subject: kernel profiling: spinlock_exit consumes 36% CPU time. Message-ID: <004001c92871$fdec0a10$01000001@china.huawei.com> Hi, folks, I did kernel profiling when a single thread client sends UDP packets to a single thread server on the same machine. In the output kernel profile, the first few kernel functions that consumes the most CPU time are listed below: granularity: each sample hit covers 16 byte(s) for 0.01% of 25.68 seconds % cumulative self self total time seconds seconds calls ms/call ms/call name 42.4 10.88 10.88 0 100.00% __mcount [1] 36.1 20.14 9.26 17937541 0.00 0.00 spinlock_exit [4] 4.2 21.22 1.08 3145728 0.00 0.00 in_cksum_skip [40] 1.8 21.68 0.45 7351987 0.00 0.00 generic_copyin [43] 1.1 21.96 0.29 3146028 0.00 0.00 generic_copyout [48] 1.0 22.21 0.24 2108904 0.00 0.00 Xint0x80_syscall [3] 0.8 22.42 0.21 6292131 0.00 0.00 uma_zalloc_arg [46] 0.8 22.62 0.20 1048576 0.00 0.00 soreceive_generic [9] 0.7 22.80 0.19 3145852 0.00 0.00 free [47] 0.6 22.96 0.15 6292172 0.00 0.00 uma_zfree_arg [52] 0.6 23.10 0.14 5243413 0.00 0.00 generic_bzero [53] 0.5 23.23 0.14 1048581 0.00 0.00 ip_output [23] 0.5 23.36 0.13 4221855 0.00 0.00 generic_bcopy [57] 0.4 23.47 0.11 36865859 0.00 0.00 critical_enter [61] 0.4 23.57 0.10 36865859 0.00 0.00 critical_exit [62] 0.4 23.67 0.09 17937541 0.00 0.00 spinlock_enter [63] 0.4 23.76 0.09 1048582 0.00 0.00 udp_input [21] 0.3 23.85 0.09 2108904 0.00 0.00 syscall [5] 0.3 23.93 0.08 1048587 0.00 0.00 ip_input [20] 0.3 24.00 0.07 2097156 0.00 0.00 getsock [65] 0.3 24.07 0.07 1048576 0.00 0.00 udp_send [22] It is very strange that spinlock_exit consumes over 36% CPU time while it seems a very simple function. For clarity, I paste the code of spinlock_exit here: void spinlock_exit(void) { struct thread *td; td = curthread; critical_exit(); td->td_md.md_spinlock_count--; if (td->td_md.md_spinlock_count == 0) intr_restore(td->td_md.md_saved_flags); } Since critical_exit consumes only 0.4% CPU time, does this mean the rest of spinlock_exit consume ~36% CPU time? Am I missing something? Could anybody help me understand this? Many thanks. BTW, the kernel is compiled with SMP and PREEMPTION disabled. The scheduler is ULE. Best regards, Qiu Jian From koitsu at FreeBSD.org Tue Oct 7 12:13:51 2008 From: koitsu at FreeBSD.org (Jeremy Chadwick) Date: Tue Oct 7 12:13:58 2008 Subject: kernel profiling: spinlock_exit consumes 36% CPU time. In-Reply-To: <004001c92871$fdec0a10$01000001@china.huawei.com> References: <004001c92871$fdec0a10$01000001@china.huawei.com> Message-ID: <20081007115748.GA48154@icarus.home.lan> On Tue, Oct 07, 2008 at 07:44:00PM +0800, ???? wrote: > Hi, folks, > > I did kernel profiling when a single thread client sends UDP packets to a > single thread server on the same machine. > > In the output kernel profile, the first few kernel functions that consumes > the most CPU time are listed below: > > granularity: each sample hit covers 16 byte(s) for 0.01% of 25.68 seconds > > % cumulative self self total > time seconds seconds calls ms/call ms/call name > 42.4 10.88 10.88 0 100.00% __mcount [1] > 36.1 20.14 9.26 17937541 0.00 0.00 spinlock_exit [4] > 4.2 21.22 1.08 3145728 0.00 0.00 in_cksum_skip [40] > 1.8 21.68 0.45 7351987 0.00 0.00 generic_copyin [43] > 1.1 21.96 0.29 3146028 0.00 0.00 generic_copyout [48] > 1.0 22.21 0.24 2108904 0.00 0.00 Xint0x80_syscall [3] > 0.8 22.42 0.21 6292131 0.00 0.00 uma_zalloc_arg [46] > 0.8 22.62 0.20 1048576 0.00 0.00 soreceive_generic [9] > 0.7 22.80 0.19 3145852 0.00 0.00 free [47] > 0.6 22.96 0.15 6292172 0.00 0.00 uma_zfree_arg [52] > 0.6 23.10 0.14 5243413 0.00 0.00 generic_bzero [53] > 0.5 23.23 0.14 1048581 0.00 0.00 ip_output [23] > 0.5 23.36 0.13 4221855 0.00 0.00 generic_bcopy [57] > 0.4 23.47 0.11 36865859 0.00 0.00 critical_enter [61] > 0.4 23.57 0.10 36865859 0.00 0.00 critical_exit [62] > 0.4 23.67 0.09 17937541 0.00 0.00 spinlock_enter [63] > 0.4 23.76 0.09 1048582 0.00 0.00 udp_input [21] > 0.3 23.85 0.09 2108904 0.00 0.00 syscall [5] > 0.3 23.93 0.08 1048587 0.00 0.00 ip_input [20] > 0.3 24.00 0.07 2097156 0.00 0.00 getsock [65] > 0.3 24.07 0.07 1048576 0.00 0.00 udp_send [22] > > It is very strange that spinlock_exit consumes over 36% CPU time while it > seems a very simple function. > > For clarity, I paste the code of spinlock_exit here: > > void > spinlock_exit(void) > { > struct thread *td; > > td = curthread; > critical_exit(); > td->td_md.md_spinlock_count--; > if (td->td_md.md_spinlock_count == 0) > intr_restore(td->td_md.md_saved_flags); > } > > Since critical_exit consumes only 0.4% CPU time, does this mean the rest of > spinlock_exit consume ~36% CPU time? > > Am I missing something? Could anybody help me understand this? Many thanks. > > BTW, the kernel is compiled with SMP and PREEMPTION disabled. The scheduler > is ULE. What FreeBSD version, and what build date of the kernel? -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From julian at elischer.org Tue Oct 7 13:28:07 2008 From: julian at elischer.org (Julian Elischer) Date: Tue Oct 7 13:28:20 2008 Subject: kernel profiling: spinlock_exit consumes 36% CPU time. In-Reply-To: <004001c92871$fdec0a10$01000001@china.huawei.com> References: <004001c92871$fdec0a10$01000001@china.huawei.com> Message-ID: <48EB63E3.60604@elischer.org> ?? wrote: > Hi, folks, [...] spinlocks disable interrupts so the profiling interrupt is held off from the moment that the spinlock is entered to the moment it is exited, and all of that time is attributed to spinlock_exit(). so that this tells you that 3% of your time is spent under spinlocks which is a lot. as others have asked, "what version"? you should look up lock profiling to see WHICH lock is teh ine in question. From jhb at freebsd.org Tue Oct 7 14:04:17 2008 From: jhb at freebsd.org (John Baldwin) Date: Tue Oct 7 14:04:34 2008 Subject: kernel profiling: spinlock_exit consumes 36% CPU time. In-Reply-To: <004001c92871$fdec0a10$01000001@china.huawei.com> References: <004001c92871$fdec0a10$01000001@china.huawei.com> Message-ID: <200810070938.04673.jhb@freebsd.org> On Tuesday 07 October 2008 07:44:00 am ?? wrote: > Hi, folks, > > I did kernel profiling when a single thread client sends UDP packets to a > single thread server on the same machine. > > In the output kernel profile, the first few kernel functions that consumes > the most CPU time are listed below: > > granularity: each sample hit covers 16 byte(s) for 0.01% of 25.68 seconds > > % cumulative self self total > time seconds seconds calls ms/call ms/call name > 42.4 10.88 10.88 0 100.00% __mcount [1] > 36.1 20.14 9.26 17937541 0.00 0.00 spinlock_exit [4] > 4.2 21.22 1.08 3145728 0.00 0.00 in_cksum_skip [40] > 1.8 21.68 0.45 7351987 0.00 0.00 generic_copyin [43] > 1.1 21.96 0.29 3146028 0.00 0.00 generic_copyout [48] > 1.0 22.21 0.24 2108904 0.00 0.00 Xint0x80_syscall [3] > 0.8 22.42 0.21 6292131 0.00 0.00 uma_zalloc_arg [46] > 0.8 22.62 0.20 1048576 0.00 0.00 soreceive_generic [9] > > It is very strange that spinlock_exit consumes over 36% CPU time while it > seems a very simple function. It's because the intr_restore() re-enables interrupts and the resulting time spent executing the handlers for any pending interrupts are attributed to spinlock_exit(). -- John Baldwin From jhein at timing.com Wed Oct 8 04:45:24 2008 From: jhein at timing.com (John Hein) Date: Wed Oct 8 04:45:36 2008 Subject: (no subject) Message-ID: <18668.10429.689076.595798@gromit.timing.com> In June pthread_cleanup_push & pthread_cleanup_pop were changed to macros that look like so... +#define pthread_cleanup_push(cleanup_routine, cleanup_arg) \ + { \ + struct _pthread_cleanup_info __cleanup_info__; \ + __pthread_cleanup_push_imp(cleanup_routine, cleanup_arg,\ + &__cleanup_info__); \ + { + +#define pthread_cleanup_pop(execute) \ + } \ + __pthread_cleanup_pop_imp(execute); \ + } + This breaks code where the pop is done in an inner block level. simplified example: pthread_cleanup_push(docleanup, &foo); try { dostuff(); } catch (...) { pthread_cleanup_pop(1); throw; } pthread_cleanup_pop(1); Does anyone have a recommended fix? From jhein at timing.com Wed Oct 8 04:45:25 2008 From: jhein at timing.com (John Hein) Date: Wed Oct 8 04:45:36 2008 Subject: pthread_cleanup_push & pthread_cleanup_pop usage Message-ID: <18668.10465.699531.162573@gromit.timing.com> In June pthread_cleanup_push & pthread_cleanup_pop were changed to macros that look like so... +#define pthread_cleanup_push(cleanup_routine, cleanup_arg) \ + { \ + struct _pthread_cleanup_info __cleanup_info__; \ + __pthread_cleanup_push_imp(cleanup_routine, cleanup_arg,\ + &__cleanup_info__); \ + { + +#define pthread_cleanup_pop(execute) \ + } \ + __pthread_cleanup_pop_imp(execute); \ + } + This breaks code where the pop is done in an inner block level. simplified example: pthread_cleanup_push(docleanup, &foo); try { dostuff(); } catch (...) { pthread_cleanup_pop(1); throw; } pthread_cleanup_pop(1); Does anyone have a recommended fix? From alfred at freebsd.org Wed Oct 8 05:12:16 2008 From: alfred at freebsd.org (Alfred Perlstein) Date: Wed Oct 8 05:12:46 2008 Subject: pthread_cleanup_push & pthread_cleanup_pop usage In-Reply-To: <18668.10465.699531.162573@gromit.timing.com> References: <18668.10465.699531.162573@gromit.timing.com> Message-ID: <20081008045447.GY36572@elvis.mu.org> * John Hein [081007 21:45] wrote: > In June pthread_cleanup_push & pthread_cleanup_pop were changed to macros > that look like so... Hey John, I found the same problem when working on QNX a while back, however that is really how it's supposed to be set up. I would suggest the following construct to fix the problem, make your own per-thread stack of destructors that are callable as functions and not macros. It's not too hard to do. Just use a pthread_key and pthread_once thingy to write a library to do it, shouldn't take more than a hundred lines of code. FWIW, OS X and QNX have the same set of macros, not sure about other OSes. -Alfred > > +#define pthread_cleanup_push(cleanup_routine, cleanup_arg) \ > + { \ > + struct _pthread_cleanup_info __cleanup_info__; \ > + __pthread_cleanup_push_imp(cleanup_routine, cleanup_arg,\ > + &__cleanup_info__); \ > + { > + > +#define pthread_cleanup_pop(execute) \ > + } \ > + __pthread_cleanup_pop_imp(execute); \ > + } > + > > > This breaks code where the pop is done in an inner block level. > > simplified example: > > pthread_cleanup_push(docleanup, &foo); > try > { > dostuff(); > } > catch (...) > { > pthread_cleanup_pop(1); > throw; > } > pthread_cleanup_pop(1); > > Does anyone have a recommended fix? > _______________________________________________ > freebsd-threads@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-threads > To unsubscribe, send any mail to "freebsd-threads-unsubscribe@freebsd.org" -- - Alfred Perlstein From deischen at freebsd.org Wed Oct 8 06:31:04 2008 From: deischen at freebsd.org (Daniel Eischen) Date: Wed Oct 8 06:31:11 2008 Subject: pthread_cleanup_push & pthread_cleanup_pop usage In-Reply-To: <20081008045447.GY36572@elvis.mu.org> References: <18668.10465.699531.162573@gromit.timing.com> <20081008045447.GY36572@elvis.mu.org> Message-ID: On Tue, 7 Oct 2008, Alfred Perlstein wrote: > * John Hein [081007 21:45] wrote: >> In June pthread_cleanup_push & pthread_cleanup_pop were changed to macros >> that look like so... > > Hey John, I found the same problem when working on QNX a while back, > however that is really how it's supposed to be set up. > > I would suggest the following construct to fix the problem, > make your own per-thread stack of destructors that are callable > as functions and not macros. > > It's not too hard to do. > > Just use a pthread_key and pthread_once thingy to write a library > to do it, shouldn't take more than a hundred lines of code. > > FWIW, OS X and QNX have the same set of macros, not sure about > other OSes. Solaris as well. Just conditionally undef them before you use them. #ifdef pthread_cleanup_push #undef pthread_cleanup_push #endif #ifdef pthread_cleanup_pop #undef pthread_cleanup_pop #endif The library versions are still there (they have to be in order to be callable from non-C/C++ languages). -- DE From qj at huawei.com Wed Oct 8 07:13:10 2008 From: qj at huawei.com (=?gb2312?B?x/G9ow==?=) Date: Wed Oct 8 07:13:27 2008 Subject: kernel profiling: spinlock_exit consumes 36% CPU time. In-Reply-To: <20081007115748.GA48154@icarus.home.lan> Message-ID: <000301c92915$1cd0ae60$01000001@china.huawei.com> Forgot to meantion that the test is based on FreeBSD kernel 7.0 2000807 snapshot. The kernel was compiled with a modified version of GENERIC configuration. With SMP and PREEMPTION disabled and kernel profiling enabled. -----Original Message----- From: Jeremy Chadwick [mailto:koitsu@FreeBSD.org] Sent: Tuesday, October 07, 2008 7:58 PM To: ???? Cc: freebsd-questions@freebsd.org; freebsd-net@FreeBSD.org; freebsd-threads@freebsd.org Subject: Re: kernel profiling: spinlock_exit consumes 36% CPU time. On Tue, Oct 07, 2008 at 07:44:00PM +0800, ???? wrote: > Hi, folks, > > I did kernel profiling when a single thread client sends UDP packets > to a single thread server on the same machine. > > In the output kernel profile, the first few kernel functions that > consumes the most CPU time are listed below: > > granularity: each sample hit covers 16 byte(s) for 0.01% of 25.68 > seconds > > % cumulative self self total > time seconds seconds calls ms/call ms/call name > 42.4 10.88 10.88 0 100.00% __mcount [1] > 36.1 20.14 9.26 17937541 0.00 0.00 spinlock_exit [4] > 4.2 21.22 1.08 3145728 0.00 0.00 in_cksum_skip [40] > 1.8 21.68 0.45 7351987 0.00 0.00 generic_copyin [43] > 1.1 21.96 0.29 3146028 0.00 0.00 generic_copyout [48] > 1.0 22.21 0.24 2108904 0.00 0.00 Xint0x80_syscall [3] > 0.8 22.42 0.21 6292131 0.00 0.00 uma_zalloc_arg [46] > 0.8 22.62 0.20 1048576 0.00 0.00 soreceive_generic [9] > 0.7 22.80 0.19 3145852 0.00 0.00 free [47] > 0.6 22.96 0.15 6292172 0.00 0.00 uma_zfree_arg [52] > 0.6 23.10 0.14 5243413 0.00 0.00 generic_bzero [53] > 0.5 23.23 0.14 1048581 0.00 0.00 ip_output [23] > 0.5 23.36 0.13 4221855 0.00 0.00 generic_bcopy [57] > 0.4 23.47 0.11 36865859 0.00 0.00 critical_enter [61] > 0.4 23.57 0.10 36865859 0.00 0.00 critical_exit [62] > 0.4 23.67 0.09 17937541 0.00 0.00 spinlock_enter [63] > 0.4 23.76 0.09 1048582 0.00 0.00 udp_input [21] > 0.3 23.85 0.09 2108904 0.00 0.00 syscall [5] > 0.3 23.93 0.08 1048587 0.00 0.00 ip_input [20] > 0.3 24.00 0.07 2097156 0.00 0.00 getsock [65] > 0.3 24.07 0.07 1048576 0.00 0.00 udp_send [22] > > It is very strange that spinlock_exit consumes over 36% CPU time while > it seems a very simple function. > > For clarity, I paste the code of spinlock_exit here: > > void > spinlock_exit(void) > { > struct thread *td; > > td = curthread; > critical_exit(); > td->td_md.md_spinlock_count--; > if (td->td_md.md_spinlock_count == 0) > intr_restore(td->td_md.md_saved_flags); > } > > Since critical_exit consumes only 0.4% CPU time, does this mean the > rest of spinlock_exit consume ~36% CPU time? > > Am I missing something? Could anybody help me understand this? Many thanks. > > BTW, the kernel is compiled with SMP and PREEMPTION disabled. The > scheduler is ULE. What FreeBSD version, and what build date of the kernel? -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From jhb at freebsd.org Wed Oct 8 19:11:02 2008 From: jhb at freebsd.org (John Baldwin) Date: Wed Oct 8 19:11:09 2008 Subject: kernel profiling: spinlock_exit consumes 36% CPU time. In-Reply-To: <000a01c9291a$b81fa560$01000001@china.huawei.com> References: <000a01c9291a$b81fa560$01000001@china.huawei.com> Message-ID: <200810081116.10298.jhb@freebsd.org> On Wednesday 08 October 2008 03:51:48 am ?? wrote: > Many thanks for the information. > > Could we say that interrupt handlers consumed ~36% execution time? > > Is this number too high? Is it possible that we abuse the use of critical > sections in kernel? I think whether or not it is high depends on the workload. -- John Baldwin From jhb at freebsd.org Wed Oct 8 19:11:17 2008 From: jhb at freebsd.org (John Baldwin) Date: Wed Oct 8 19:11:30 2008 Subject: pthread_cleanup_push & pthread_cleanup_pop usage In-Reply-To: <18668.10465.699531.162573@gromit.timing.com> References: <18668.10465.699531.162573@gromit.timing.com> Message-ID: <200810081409.31822.jhb@freebsd.org> On Tuesday 07 October 2008 11:28:33 pm John Hein wrote: > In June pthread_cleanup_push & pthread_cleanup_pop were changed to macros > that look like so... > > +#define pthread_cleanup_push(cleanup_routine, cleanup_arg) \ > + { \ > + struct _pthread_cleanup_info __cleanup_info__; \ > + __pthread_cleanup_push_imp(cleanup_routine, cleanup_arg,\ > + &__cleanup_info__); \ > + { > + > +#define pthread_cleanup_pop(execute) \ > + } \ > + __pthread_cleanup_pop_imp(execute); \ > + } > + > > > This breaks code where the pop is done in an inner block level. > > simplified example: > > pthread_cleanup_push(docleanup, &foo); > try > { > dostuff(); > } > catch (...) > { > pthread_cleanup_pop(1); > throw; > } > pthread_cleanup_pop(1); > > Does anyone have a recommended fix? Don't Do That (tm). The spec explicitly states that they can be defined as macros and that code using it needs to put them in the same block. -- John Baldwin From brde at optusnet.com.au Wed Oct 8 19:32:40 2008 From: brde at optusnet.com.au (Bruce Evans) Date: Wed Oct 8 19:32:47 2008 Subject: kernel profiling: spinlock_exit consumes 36% CPU time. In-Reply-To: <200810070938.04673.jhb@freebsd.org> References: <004001c92871$fdec0a10$01000001@china.huawei.com> <200810070938.04673.jhb@freebsd.org> Message-ID: <20081008210104.S20625@delplex.bde.org> On Tue, 7 Oct 2008, John Baldwin wrote: > On Tuesday 07 October 2008 07:44:00 am Ηρ½£ wrote: >> Hi, folks, >> >> I did kernel profiling when a single thread client sends UDP packets to a >> single thread server on the same machine. >> >> In the output kernel profile, the first few kernel functions that consumes >> the most CPU time are listed below: >> >> granularity: each sample hit covers 16 byte(s) for 0.01% of 25.68 seconds >> >> % cumulative self self total >> time seconds seconds calls ms/call ms/call name >> 42.4 10.88 10.88 0 100.00% __mcount [1] >> 36.1 20.14 9.26 17937541 0.00 0.00 spinlock_exit [4] >> 4.2 21.22 1.08 3145728 0.00 0.00 in_cksum_skip [40] >> 1.8 21.68 0.45 7351987 0.00 0.00 generic_copyin [43] >> 1.1 21.96 0.29 3146028 0.00 0.00 generic_copyout [48] >> 1.0 22.21 0.24 2108904 0.00 0.00 Xint0x80_syscall [3] >> 0.8 22.42 0.21 6292131 0.00 0.00 uma_zalloc_arg [46] >> 0.8 22.62 0.20 1048576 0.00 0.00 soreceive_generic [9] >> >> It is very strange that spinlock_exit consumes over 36% CPU time while it >> seems a very simple function. > > It's because the intr_restore() re-enables interrupts and the resulting time > spent executing the handlers for any pending interrupts are attributed to > spinlock_exit(). This is one of many defects that are not present in high resolution kernel profiling (kgmon -B instead of kgmon -b; availaible on amd64 and i386). However, high resolution kernel profiling doesn't work right with SMP, and was completely broken by gcc-4. Ordinary profiling was less completely broken by gcc-4, and you can recover the old behaviour by turning off new optimizations (mainly -funit-at-a-time and/or -finline-functions-called-once and or all of -O2). Bruce From alfred at freebsd.org Wed Oct 8 19:53:25 2008 From: alfred at freebsd.org (Alfred Perlstein) Date: Wed Oct 8 19:53:31 2008 Subject: pthread_cleanup_push & pthread_cleanup_pop usage In-Reply-To: <18669.3729.459915.769388@gromit.timing.com> References: <18668.10465.699531.162573@gromit.timing.com> <200810081409.31822.jhb@freebsd.org> <18669.3729.459915.769388@gromit.timing.com> Message-ID: <20081008195324.GM36572@elvis.mu.org> * John Hein [081008 12:48] wrote: > John Baldwin wrote at 14:09 -0400 on Oct 8, 2008: > > Don't Do That (tm). The spec explicitly states that they can be defined as > > macros and that code using it needs to put them in the same block. > > Thanks for the responses. I like deischen's response best to work > around the problem short term, but I suspect that's not very portable. > And as jhb describes using it outside the same block violates the > spirit of intended use. > > Note that the example I gave (pop off the cleanup stack in a catch > block) is not our only "violation". We also push in one function and > either never [explicitly] pop or possibly pop in another function. > > So that leaves me wondering about possible techniques for cleanup on > thread cancellation or exit. Alfred touched on one method, and I see > that pthread_key_create takes an optional destructor, so that sounds > like a possible avenue to explore. I'm positive that you'll have a solution given my starting point within a few hours. :) All the cpp_cleanup_push() needs to do is to: pthread_once() -> set up the pthread_key and destructor as you said. then pthread_getspecific to get the stack if no stack, make one and set it. then push your cleanup function. good luck, -- - Alfred Perlstein From deischen at freebsd.org Wed Oct 8 20:40:31 2008 From: deischen at freebsd.org (Daniel Eischen) Date: Wed Oct 8 20:40:37 2008 Subject: pthread_cleanup_push & pthread_cleanup_pop usage In-Reply-To: <20081008195324.GM36572@elvis.mu.org> References: <18668.10465.699531.162573@gromit.timing.com> <200810081409.31822.jhb@freebsd.org> <18669.3729.459915.769388@gromit.timing.com> <20081008195324.GM36572@elvis.mu.org> Message-ID: On Wed, 8 Oct 2008, Alfred Perlstein wrote: > * John Hein [081008 12:48] wrote: >> John Baldwin wrote at 14:09 -0400 on Oct 8, 2008: >> > Don't Do That (tm). The spec explicitly states that they can be defined as >> > macros and that code using it needs to put them in the same block. >> >> Thanks for the responses. I like deischen's response best to work >> around the problem short term, but I suspect that's not very portable. It's probably more portable than you think. The library calls really need to be there so they can be linked to from non-C languages. >> And as jhb describes using it outside the same block violates the >> spirit of intended use. >> >> Note that the example I gave (pop off the cleanup stack in a catch >> block) is not our only "violation". We also push in one function and >> either never [explicitly] pop or possibly pop in another function. >> >> So that leaves me wondering about possible techniques for cleanup on >> thread cancellation or exit. Alfred touched on one method, and I see >> that pthread_key_create takes an optional destructor, so that sounds >> like a possible avenue to explore. > > I'm positive that you'll have a solution given my starting point > within a few hours. :) > > All the cpp_cleanup_push() needs to do is to: > > pthread_once() -> set up the pthread_key and destructor as you said. > then pthread_getspecific to get the stack > if no stack, make one and set it. > then push your cleanup function. > > good luck, -- DE From jhein at timing.com Wed Oct 8 20:45:25 2008 From: jhein at timing.com (John Hein) Date: Wed Oct 8 20:45:32 2008 Subject: pthread_cleanup_push & pthread_cleanup_pop usage In-Reply-To: <200810081409.31822.jhb@freebsd.org> References: <18668.10465.699531.162573@gromit.timing.com> <200810081409.31822.jhb@freebsd.org> Message-ID: <18669.3729.459915.769388@gromit.timing.com> John Baldwin wrote at 14:09 -0400 on Oct 8, 2008: > Don't Do That (tm). The spec explicitly states that they can be defined as > macros and that code using it needs to put them in the same block. Thanks for the responses. I like deischen's response best to work around the problem short term, but I suspect that's not very portable. And as jhb describes using it outside the same block violates the spirit of intended use. Note that the example I gave (pop off the cleanup stack in a catch block) is not our only "violation". We also push in one function and either never [explicitly] pop or possibly pop in another function. So that leaves me wondering about possible techniques for cleanup on thread cancellation or exit. Alfred touched on one method, and I see that pthread_key_create takes an optional destructor, so that sounds like a possible avenue to explore. From davidxu at freebsd.org Thu Oct 9 01:24:09 2008 From: davidxu at freebsd.org (David Xu) Date: Thu Oct 9 01:24:19 2008 Subject: pthread_cleanup_push & pthread_cleanup_pop usage In-Reply-To: <18668.10465.699531.162573@gromit.timing.com> References: <18668.10465.699531.162573@gromit.timing.com> Message-ID: <48ED5DA4.9060803@freebsd.org> John Hein wrote: > In June pthread_cleanup_push & pthread_cleanup_pop were changed to macros > that look like so... > > +#define pthread_cleanup_push(cleanup_routine, cleanup_arg) \ > + { \ > + struct _pthread_cleanup_info __cleanup_info__; \ > + __pthread_cleanup_push_imp(cleanup_routine, cleanup_arg,\ > + &__cleanup_info__); \ > + { > + > +#define pthread_cleanup_pop(execute) \ > + } \ > + __pthread_cleanup_pop_imp(execute); \ > + } > + > > > This breaks code where the pop is done in an inner block level. > > simplified example: > > pthread_cleanup_push(docleanup, &foo); > try > { > dostuff(); > } > catch (...) > { > pthread_cleanup_pop(1); > throw; > } > pthread_cleanup_pop(1); > > Does anyone have a recommended fix? Your problem can be fixed as deischen@ said. but I think it can be resolved by stack unwinding code I am working in progress, when a C++ exception is thrown, onstack C++ objects along with pthread cleanup handlers will be invoked automatically. Regards, David Xu From bugmaster at FreeBSD.org Mon Oct 13 11:06:58 2008 From: bugmaster at FreeBSD.org (FreeBSD bugmaster) Date: Mon Oct 13 11:09:10 2008 Subject: Current problem reports assigned to freebsd-threads@FreeBSD.org Message-ID: <200810131106.m9DB6vAn029590@freefall.freebsd.org> Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o threa/127225 threads bug in lib/libthr/thread/thr_init.c o threa/126950 threads [patch] rtld(1): rtld malloc is thread-unsafe o kern/126128 threads [patch] pthread_condattr_getpshared is broken o threa/122923 threads 'nice' does not prevent background process from steali o threa/121336 threads lang/neko threading ok on UP, broken on SMP (FreeBSD 7 o threa/118715 threads kse problem o threa/116668 threads can no longer use jdk15 with libthr on -stable SMP o threa/116181 threads /dev/io-related io access permissions are not propagat o threa/115211 threads pthread_atfork misbehaves in initial thread o threa/110636 threads [request] gdb(1): using gdb with multi thread applicat o threa/110306 threads apache 2.0 segmentation violation when calling gethost o threa/103975 threads Implicit loading/unloading of libpthread.so may crash o threa/101323 threads fork(2) in threaded programs broken. s threa/100815 threads FBSD 5.5 broke nanosleep in libc_r s threa/94467 threads send(), sendto() and sendmsg() are not correct in libc s threa/84483 threads problems with devel/nspr and -lc_r on 4.x o threa/83914 threads [libc] popen() doesn't work in static threaded program o threa/80992 threads abort() sometimes not caught by gdb depending on threa o threa/80435 threads panic on high loads o threa/79887 threads [patch] freopen() isn't thread-safe o threa/79683 threads svctcp_create() fails if multiple threads call at the s threa/76694 threads fork cause hang in dup()/close() function in child (-l s threa/76690 threads fork hang in child for -lc_r o threa/75374 threads pthread_kill() ignores SA_SIGINFO flag o threa/75273 threads FBSD 5.3 libpthread (KSE) bug o threa/72953 threads fork() unblocks blocked signals w/o PTHREAD_SCOPE_SYST o threa/70975 threads [sysvipc] unexpected and unreliable behaviour when usi s threa/69020 threads pthreads library leaks _gc_mutex s threa/49087 threads Signals lost in programs linked with libc_r s threa/48856 threads Setting SIGCHLD to SIG_IGN still leaves zombies under s threa/40671 threads pthread_cancel doesn't remove thread from condition qu s threa/39922 threads [threads] [patch] Threaded applications executed with s threa/37676 threads libc_r: msgsnd(), msgrcv(), pread(), pwrite() need wra s threa/34536 threads accept() blocks other threads s kern/32295 threads [libc_r] [patch] pthread(3) dont dequeue signals s threa/30464 threads pthread mutex attributes -- pshared s threa/24632 threads libc_r delicate deviation from libc in handling SIGCHL s threa/24472 threads libc_r does not honor SO_SNDTIMEO/SO_RCVTIMEO socket o 38 problems total. From davidxu at freebsd.org Thu Oct 16 01:44:40 2008 From: davidxu at freebsd.org (David Xu) Date: Thu Oct 16 01:45:02 2008 Subject: pthread_cleanup_push & pthread_cleanup_pop usage In-Reply-To: References: <18668.10465.699531.162573@gromit.timing.com> <20081008045447.GY36572@elvis.mu.org> Message-ID: <48F69CF4.9050905@freebsd.org> Daniel Eischen wrote: > On Tue, 7 Oct 2008, Alfred Perlstein wrote: > >> * John Hein [081007 21:45] wrote: >>> In June pthread_cleanup_push & pthread_cleanup_pop were changed to >>> macros >>> that look like so... >> >> Hey John, I found the same problem when working on QNX a while back, >> however that is really how it's supposed to be set up. >> >> I would suggest the following construct to fix the problem, >> make your own per-thread stack of destructors that are callable >> as functions and not macros. >> >> It's not too hard to do. >> >> Just use a pthread_key and pthread_once thingy to write a library >> to do it, shouldn't take more than a hundred lines of code. >> >> FWIW, OS X and QNX have the same set of macros, not sure about >> other OSes. > > Solaris as well. > > Just conditionally undef them before you use them. > > #ifdef pthread_cleanup_push > #undef pthread_cleanup_push > #endif > #ifdef pthread_cleanup_pop > #undef pthread_cleanup_pop > #endif > > The library versions are still there (they have to be in order > to be callable from non-C/C++ languages). > One of possible solutions is we define a C++ class in pthread.h: #ifdef __cplusplus class __pthread_cleanup_obj { void (*__f)(void *); void *__a; int __execeute; public: __pthread_cleanup_obj(void (*__cleanup_routine)(void *), void *__arg) { f = __cleanup_routine; a = __arg; __execute = 0; } ~__pthread_cleanup_obj() { if (__execute) __f(__a); } void __set_execute(int __e) { __execute = __e; } }; #define pthread_cleanup_push(f, a) { \ __pthread_cleanup_obj __cleanup(f, a); \ { #define pthread_cleanup_pop(e) \ __cleanup.__set_execute(e); \ } \ } #endif but because there is no specification for C++ and threading, it is unknown which behavior is desired. David Xu From kurt at intricatesoftware.com Fri Oct 17 16:50:02 2008 From: kurt at intricatesoftware.com (Kurt Miller) Date: Fri Oct 17 16:50:10 2008 Subject: threads/128180: pthread_cond_broadcast() lost wakup Message-ID: <200810171640.m9HGexJ1090893@www.freebsd.org> >Number: 128180 >Category: threads >Synopsis: pthread_cond_broadcast() lost wakup >Confidential: no >Severity: serious >Priority: high >Responsible: freebsd-threads >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Fri Oct 17 16:50:01 UTC 2008 >Closed-Date: >Last-Modified: >Originator: Kurt Miller >Release: 6.3-RELEASE >Organization: Intricate Software >Environment: FreeBSD fbsd-amd64-63.intricatesoftware.com 6.3-RELEASE FreeBSD 6.3-RELEASE #0: Wed Jan 16 01:43:02 UTC 2008 root@palmer.cse.buffalo.edu:/usr/obj/usr/src/sys/SMP amd64 >Description: I've been investigating a deadlock in the jvm that occurs with the concurrent mark sweep garbage collector. The cause appears to be due to the kernel failing to wake up all threads waiting on a condition variable. I have written a test program that mimics the jvm's underlying pattern. It reproduces the deadlock quickly and exhibits the same problem. The general idea is that one thread sends a broadcast to a group of worker threads. The worker threads perform some tasks, coordinate their completion and broadcast on the same condition variable they are done. The design is a bit heavy on the use of the one condition variable, however it does appear to be valid if not ideal. The deadlock occurs with the following system setup: 6.3-RELEASE SMP amd64 kernel libthr 2 or more cores I have not yet checked other releases or setups. The test program outputs periodic printf's indicating progress is being made. When it stops the process is deadlocked. The lost wakeup can be confirmed by inspecting the saved_waiters local var in main(). Each time the deadlock occurs I see that saved_waiters is 8 which tells me all eight worker threads were waiting on the condition variable when the broadcast was sent. Then switch to the thread that is still waiting on the condition variable, and you can see that the last_cycle local var is one behind the cycles global var which indicates it didn't receive the last wakeup. >How-To-Repeat: #include #include #include pthread_mutex_t group_mutex = PTHREAD_MUTEX_INITIALIZER; pthread_cond_t group_cond_var = PTHREAD_COND_INITIALIZER; volatile int tickets; volatile int waiters; volatile int finished; int term_count; volatile unsigned long cycles; void *thread_main(void * thread_num); #define NTHREADS 8 #define NYIELDS 1000 inline void atomicinc(volatile int* val) { __asm__ __volatile__ ("lock addl $1,(%0)" : : "r" (val) : "cc", "memory"); } int main( int argc, char *argv[] ) { long t_num; pthread_t tid[NTHREADS]; volatile int saved_waiters; /* startup threads */ for (t_num=0; t_num < NTHREADS; t_num++) { pthread_create( &tid[t_num], NULL, thread_main, (void *)t_num ); } for(;;) { /* monitor progress on stdout */ if (cycles % 5000 == 0) printf("cycles %lu\n", cycles); /* broadcast to workers to work */ pthread_mutex_lock(&group_mutex); cycles++; term_count = 0; finished = 0; tickets=NTHREADS; saved_waiters = waiters; pthread_cond_broadcast(&group_cond_var); pthread_mutex_unlock(&group_mutex); /* wait for workers to finish */ pthread_mutex_lock(&group_mutex); while (finished != NTHREADS) pthread_cond_wait(&group_cond_var, &group_mutex); pthread_mutex_unlock(&group_mutex); } return 0; } void * thread_main(void *thread_num) { unsigned long yield_count=0; unsigned long sleep_count=0; u_int32_t i, busy_loop = arc4random() & 0x7FFF; u_int32_t dummy = busy_loop; pthread_cond_t sleep_cond_var; pthread_mutex_t sleep_mutex; struct timeval tmptime; struct timeval delay = {0, 1}; struct timespec waketime; volatile unsigned long last_cycle; pthread_mutex_init(&sleep_mutex, NULL); pthread_cond_init(&sleep_cond_var, NULL); for (;;) { pthread_mutex_lock(&group_mutex); waiters++; while (tickets == 0) pthread_cond_wait(&group_cond_var, &group_mutex); waiters--; tickets--; last_cycle = cycles; pthread_mutex_unlock(&group_mutex); /* do something busy */ for (i = 0; i < busy_loop; i++) dummy *= i; /* sync termination */ atomicinc(&term_count); for(;;) { if (term_count == NTHREADS) break; if (yield_count < NYIELDS) { yield_count++; sched_yield(); } else { yield_count = 0; sleep_count++; // 1.6 uses pthread_cond_timedwait for sleeping gettimeofday(&tmptime, NULL); timeradd(&tmptime, &delay, &tmptime); waketime.tv_sec = tmptime.tv_sec; waketime.tv_nsec = tmptime.tv_usec * 1000; pthread_mutex_lock(&sleep_mutex); pthread_cond_timedwait(&sleep_cond_var, &sleep_mutex, &waketime); pthread_mutex_unlock(&sleep_mutex); } } /* ok all terminated now let everyone know */ pthread_mutex_lock(&group_mutex); finished++; pthread_cond_broadcast(&group_cond_var); pthread_mutex_unlock(&group_mutex); } return NULL; } >Fix: >Release-Note: >Audit-Trail: >Unformatted: From kurt at intricatesoftware.com Fri Oct 17 20:00:13 2008 From: kurt at intricatesoftware.com (Kurt Miller) Date: Fri Oct 17 20:00:18 2008 Subject: threads/128180: pthread_cond_broadcast() lost wakup Message-ID: <200810172000.m9HK0CV1039353@freefall.freebsd.org> The following reply was made to PR threads/128180; it has been noted by GNATS. From: Kurt Miller To: bug-followup@FreeBSD.org Cc: Subject: Re: threads/128180: pthread_cond_broadcast() lost wakup Date: Fri, 17 Oct 2008 15:56:57 -0400 I've reproduced this on the following setup: 6.3-RELEASE SMP i386 kernel libthr 2 cores However, the lost wakeup is on the main thread. The last broadcast wakeup from the worker threads gets lost and the process deadlocks. So far the test program has not provoked the lost wakeup using libpthread on 6.3. Initial 7.0 (amd64 libthr) testing has not provoked the issue yet either. From deischen at freebsd.org Fri Oct 17 23:45:00 2008 From: deischen at freebsd.org (Daniel Eischen) Date: Fri Oct 17 23:45:07 2008 Subject: threads/128180: pthread_cond_broadcast() lost wakup In-Reply-To: <200810171640.m9HGexJ1090893@www.freebsd.org> References: <200810171640.m9HGexJ1090893@www.freebsd.org> Message-ID: On Fri, 17 Oct 2008, Kurt Miller wrote: > The test program outputs periodic printf's indicating > progress is being made. When it stops the process is > deadlocked. The lost wakeup can be confirmed by inspecting > the saved_waiters local var in main(). Each time the > deadlock occurs I see that saved_waiters is 8 which tells > me all eight worker threads were waiting on the condition > variable when the broadcast was sent. Then switch to the > thread that is still waiting on the condition variable, > and you can see that the last_cycle local var is one behind > the cycles global var which indicates it didn't receive the > last wakeup. The test program doesn't look correct to me. It seems possible for only a few of the threads (as little as 2) to do all the work. Thread 1 can start doing work, then wait for a broadcast. Thread 2 can start doing his work, then broadcast waking thread 1. I think you need separate condition variables, one to wake up the main thread when the last worker goes to sleep/finishes, and one to wake up the workers. -- DE From deischen at freebsd.org Sat Oct 18 00:00:10 2008 From: deischen at freebsd.org (Daniel Eischen) Date: Sat Oct 18 00:00:16 2008 Subject: threads/128180: pthread_cond_broadcast() lost wakup Message-ID: <200810180000.m9I009T8058526@freefall.freebsd.org> The following reply was made to PR threads/128180; it has been noted by GNATS. From: Daniel Eischen To: Kurt Miller Cc: freebsd-gnats-submit@freebsd.org, freebsd-threads@freebsd.org Subject: Re: threads/128180: pthread_cond_broadcast() lost wakup Date: Fri, 17 Oct 2008 19:44:58 -0400 (EDT) On Fri, 17 Oct 2008, Kurt Miller wrote: > The test program outputs periodic printf's indicating > progress is being made. When it stops the process is > deadlocked. The lost wakeup can be confirmed by inspecting > the saved_waiters local var in main(). Each time the > deadlock occurs I see that saved_waiters is 8 which tells > me all eight worker threads were waiting on the condition > variable when the broadcast was sent. Then switch to the > thread that is still waiting on the condition variable, > and you can see that the last_cycle local var is one behind > the cycles global var which indicates it didn't receive the > last wakeup. The test program doesn't look correct to me. It seems possible for only a few of the threads (as little as 2) to do all the work. Thread 1 can start doing work, then wait for a broadcast. Thread 2 can start doing his work, then broadcast waking thread 1. I think you need separate condition variables, one to wake up the main thread when the last worker goes to sleep/finishes, and one to wake up the workers. -- DE From kurt at intricatesoftware.com Sat Oct 18 03:00:18 2008 From: kurt at intricatesoftware.com (Kurt Miller) Date: Sat Oct 18 03:00:24 2008 Subject: threads/128180: pthread_cond_broadcast() lost wakup Message-ID: <200810180300.m9I30HqG073892@freefall.freebsd.org> The following reply was made to PR threads/128180; it has been noted by GNATS. From: Kurt Miller To: freebsd-gnats-submit@freebsd.org Cc: Subject: Re: threads/128180: pthread_cond_broadcast() lost wakup Date: Fri, 17 Oct 2008 22:54:11 -0400 Hi Daniel, Thanks for the review of the test program. On Friday 17 October 2008 7:44:58 pm Daniel Eischen wrote: > On Fri, 17 Oct 2008, Kurt Miller wrote: > > > The test program outputs periodic printf's indicating > > progress is being made. When it stops the process is > > deadlocked. The lost wakeup can be confirmed by inspecting > > the saved_waiters local var in main(). Each time the > > deadlock occurs I see that saved_waiters is 8 which tells > > me all eight worker threads were waiting on the condition > > variable when the broadcast was sent. Then switch to the > > thread that is still waiting on the condition variable, > > and you can see that the last_cycle local var is one behind > > the cycles global var which indicates it didn't receive the > > last wakeup. > > The test program doesn't look correct to me. It seems possible > for only a few of the threads (as little as 2) to do all the > work. Thread 1 can start doing work, then wait for a broadcast. > Thread 2 can start doing his work, then broadcast waking thread 1. I didn't fully describe why the design is the way it is. I understand some of the reasons why it was designed like this, but to fully understand it I would need to study the concurrent mark sweep garbage collector far more. I can explain a bit more of what I do understand. The controlling thread in jvm corresponds to the primordial thread in my test program. In the jvm the controlling thread is not in a loop. It just kicks off the worker threads and waits for them to complete, then returns back to the calling function. The jvm will create a worker thread per cpu which wait around for the controlling thread to kick them off. The garbage collection work is divided amongst them. The reason why my test program has 8 worker threads is because the problem was first reported to me on an dual quad core amd64 system. My test systems are just dual core. > I think you need separate condition variables, one to wake up > the main thread when the last worker goes to sleep/finishes, > and one to wake up the workers. Indeed. In my first attempts to reproduce the lost wakeup problem I wrote the test program with a separate condition variable for letting the main thread know when the last worker finished. However, that didn't reproduce the deadlock the jdk was experiencing. Only when I fully mimicked the underlying design of the jdk, did the deadlock get reproduced by the test program. Note that the jdk is written in C++ and abstraction it provides makes for some pretty ugly code when translated in plain C. I could make adjustments to the jvm code to introduce the second condition variable and incorporate that in future releases of the jdk. The problem is that the binary release of the jdk, Diablo, can't be changed without a new formal release process being followed. While the test program and the jdk's use of condition variables may not be ideal and somewhat unexpected, I do believe it is valid. It does work on Solaris, Linux and Windows without loosing wakeups. With the 6.4 release comming soon, it would be great if the lost wakeup problem (which is rather serious) could be looked at and fixed before 6.4 is released. Regards, -Kurt From bugmaster at FreeBSD.org Mon Oct 20 11:07:00 2008 From: bugmaster at FreeBSD.org (FreeBSD bugmaster) Date: Mon Oct 20 11:09:10 2008 Subject: Current problem reports assigned to freebsd-threads@FreeBSD.org Message-ID: <200810201106.m9KB6xC4082826@freefall.freebsd.org> Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o threa/128180 threads pthread_cond_broadcast(3) lost wakeup o threa/127225 threads bug in lib/libthr/thread/thr_init.c o threa/126950 threads [patch] rtld(1): rtld malloc is thread-unsafe o kern/126128 threads [patch] pthread_condattr_getpshared is broken o threa/122923 threads 'nice' does not prevent background process from steali o threa/121336 threads lang/neko threading ok on UP, broken on SMP (FreeBSD 7 o threa/118715 threads kse problem o threa/116668 threads can no longer use jdk15 with libthr on -stable SMP o threa/116181 threads /dev/io-related io access permissions are not propagat o threa/115211 threads pthread_atfork misbehaves in initial thread o threa/110636 threads [request] gdb(1): using gdb with multi thread applicat o threa/110306 threads apache 2.0 segmentation violation when calling gethost o threa/103975 threads Implicit loading/unloading of libpthread.so may crash o threa/101323 threads fork(2) in threaded programs broken. s threa/100815 threads FBSD 5.5 broke nanosleep in libc_r s threa/94467 threads send(), sendto() and sendmsg() are not correct in libc s threa/84483 threads problems with devel/nspr and -lc_r on 4.x o threa/83914 threads [libc] popen() doesn't work in static threaded program o threa/80992 threads abort() sometimes not caught by gdb depending on threa o threa/80435 threads panic on high loads o threa/79887 threads [patch] freopen() isn't thread-safe o threa/79683 threads svctcp_create() fails if multiple threads call at the s threa/76694 threads fork cause hang in dup()/close() function in child (-l s threa/76690 threads fork hang in child for -lc_r o threa/75374 threads pthread_kill() ignores SA_SIGINFO flag o threa/75273 threads FBSD 5.3 libpthread (KSE) bug o threa/72953 threads fork() unblocks blocked signals w/o PTHREAD_SCOPE_SYST o threa/70975 threads [sysvipc] unexpected and unreliable behaviour when usi s threa/69020 threads pthreads library leaks _gc_mutex s threa/49087 threads Signals lost in programs linked with libc_r s threa/48856 threads Setting SIGCHLD to SIG_IGN still leaves zombies under s threa/40671 threads pthread_cancel doesn't remove thread from condition qu s threa/39922 threads [threads] [patch] Threaded applications executed with s threa/37676 threads libc_r: msgsnd(), msgrcv(), pread(), pwrite() need wra s threa/34536 threads accept() blocks other threads s kern/32295 threads [libc_r] [patch] pthread(3) dont dequeue signals s threa/30464 threads pthread mutex attributes -- pshared s threa/24632 threads libc_r delicate deviation from libc in handling SIGCHL s threa/24472 threads libc_r does not honor SO_SNDTIMEO/SO_RCVTIMEO socket o 39 problems total. From nkoch at demig.de Tue Oct 21 08:13:50 2008 From: nkoch at demig.de (Norbert Koch) Date: Tue Oct 21 08:13:56 2008 Subject: 'libc_r: enter/leave_cancellation_point() Message-ID: <48FD88B1.3050106@demig.de> Hello, I was just inspecting libc_r for trying to understand some things and found this: <------------------------- --- src/lib/libc_r/uthread/uthread_cond.c 2002/05/24 04:32:28 1.33 +++ src/lib/libc_r/uthread/uthread_cond.c 2002/11/13 18:13:26 1.34 ... int -_pthread_cond_signal(pthread_cond_t * cond) +__pthread_cond_timedwait(pthread_cond_t *cond, pthread_mutex_t *mutex, + const struct timespec *abstime) +{ + int ret; + + _thread_enter_cancellation_point(); + ret = _pthread_cond_timedwait(cond, mutex, abstime); + _thread_enter_cancellation_point(); + return (ret); +} ----------------------------> Shouldn't that be _thread_leave_cancellation_point() after calling _pthread_cond_timedwait() ? What effect should I see if this is wrong? Best regards, Norbert Koch From nkoch at demig.de Tue Oct 21 08:32:15 2008 From: nkoch at demig.de (Norbert Koch) Date: Tue Oct 21 08:32:22 2008 Subject: 'libc_r: enter/leave_cancellation_point() Message-ID: <48FD88B1.3050106@demig.de> Hello, I was just inspecting libc_r for trying to understand some things and found this: <------------------------- --- src/lib/libc_r/uthread/uthread_cond.c 2002/05/24 04:32:28 1.33 +++ src/lib/libc_r/uthread/uthread_cond.c 2002/11/13 18:13:26 1.34 ... int -_pthread_cond_signal(pthread_cond_t * cond) +__pthread_cond_timedwait(pthread_cond_t *cond, pthread_mutex_t *mutex, + const struct timespec *abstime) +{ + int ret; + + _thread_enter_cancellation_point(); + ret = _pthread_cond_timedwait(cond, mutex, abstime); + _thread_enter_cancellation_point(); + return (ret); +} ----------------------------> Shouldn't that be _thread_leave_cancellation_point() after calling _pthread_cond_timedwait() ? What effect should I see if this is wrong? Best regards, Norbert Koch _______________________________________________ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" From alfred at freebsd.org Tue Oct 21 21:04:45 2008 From: alfred at freebsd.org (Alfred Perlstein) Date: Tue Oct 21 21:04:50 2008 Subject: 'libc_r: enter/leave_cancellation_point() In-Reply-To: <48FD88B1.3050106@demig.de> References: <48FD88B1.3050106@demig.de> Message-ID: <20081021210444.GB22503@elvis.mu.org> Hey Norbert, this is probably a bug, but might not be addressed because libc_r is not really supported any longer. Someone may pick it up, but I'm uncertain of that. * Norbert Koch [081021 01:32] wrote: > Hello, > > I was just inspecting libc_r for trying to understand > some things and found this: > > <------------------------- > > --- src/lib/libc_r/uthread/uthread_cond.c 2002/05/24 04:32:28 1.33 > +++ src/lib/libc_r/uthread/uthread_cond.c 2002/11/13 18:13:26 1.34 > > ... > > int > -_pthread_cond_signal(pthread_cond_t * cond) > +__pthread_cond_timedwait(pthread_cond_t *cond, pthread_mutex_t *mutex, > + const struct timespec *abstime) > +{ > + int ret; > + > + _thread_enter_cancellation_point(); > + ret = _pthread_cond_timedwait(cond, mutex, abstime); > + _thread_enter_cancellation_point(); > + return (ret); > +} > > > ----------------------------> > > Shouldn't that be _thread_leave_cancellation_point() after > calling _pthread_cond_timedwait() ? > What effect should I see if this is wrong? > > Best regards, > > Norbert Koch > > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-threads@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-threads > To unsubscribe, send any mail to "freebsd-threads-unsubscribe@freebsd.org" -- - Alfred Perlstein From bugmaster at FreeBSD.org Mon Oct 27 11:07:23 2008 From: bugmaster at FreeBSD.org (FreeBSD bugmaster) Date: Mon Oct 27 11:09:20 2008 Subject: Current problem reports assigned to freebsd-threads@FreeBSD.org Message-ID: <200810271107.m9RB7NjK002132@freefall.freebsd.org> Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o threa/128180 threads pthread_cond_broadcast(3) lost wakeup o threa/127225 threads bug in lib/libthr/thread/thr_init.c o threa/126950 threads [patch] rtld(1): rtld malloc is thread-unsafe o kern/126128 threads [patch] pthread_condattr_getpshared is broken o threa/122923 threads 'nice' does not prevent background process from steali o threa/121336 threads lang/neko threading ok on UP, broken on SMP (FreeBSD 7 o threa/118715 threads kse problem o threa/116668 threads can no longer use jdk15 with libthr on -stable SMP o threa/116181 threads /dev/io-related io access permissions are not propagat o threa/115211 threads pthread_atfork misbehaves in initial thread o threa/110636 threads [request] gdb(1): using gdb with multi thread applicat o threa/110306 threads apache 2.0 segmentation violation when calling gethost o threa/103975 threads Implicit loading/unloading of libpthread.so may crash o threa/101323 threads fork(2) in threaded programs broken. s threa/100815 threads FBSD 5.5 broke nanosleep in libc_r s threa/94467 threads send(), sendto() and sendmsg() are not correct in libc s threa/84483 threads problems with devel/nspr and -lc_r on 4.x o threa/83914 threads [libc] popen() doesn't work in static threaded program o threa/80992 threads abort() sometimes not caught by gdb depending on threa o threa/80435 threads panic on high loads o threa/79887 threads [patch] freopen() isn't thread-safe o threa/79683 threads svctcp_create() fails if multiple threads call at the s threa/76694 threads fork cause hang in dup()/close() function in child (-l s threa/76690 threads fork hang in child for -lc_r o threa/75374 threads pthread_kill() ignores SA_SIGINFO flag o threa/75273 threads FBSD 5.3 libpthread (KSE) bug o threa/72953 threads fork() unblocks blocked signals w/o PTHREAD_SCOPE_SYST o threa/70975 threads [sysvipc] unexpected and unreliable behaviour when usi s threa/69020 threads pthreads library leaks _gc_mutex s threa/49087 threads Signals lost in programs linked with libc_r s threa/48856 threads Setting SIGCHLD to SIG_IGN still leaves zombies under s threa/40671 threads pthread_cancel doesn't remove thread from condition qu s threa/39922 threads [threads] [patch] Threaded applications executed with s threa/37676 threads libc_r: msgsnd(), msgrcv(), pread(), pwrite() need wra s threa/34536 threads accept() blocks other threads s kern/32295 threads [libc_r] [patch] pthread(3) dont dequeue signals s threa/30464 threads pthread mutex attributes -- pshared s threa/24632 threads libc_r delicate deviation from libc in handling SIGCHL s threa/24472 threads libc_r does not honor SO_SNDTIMEO/SO_RCVTIMEO socket o 39 problems total. From sbartnikowski at barkinglizards.com Tue Oct 28 16:19:29 2008 From: sbartnikowski at barkinglizards.com (Stephen Bartnikowski) Date: Tue Oct 28 16:19:35 2008 Subject: deadlock in fork call? Message-ID: <5B0A3FB3D06E41849325AD3EA98CAD55@OCTOBER> Hi guys, To start, I'm not reporting a bug here. I'm trying to get an idea if I'm on the right track. Also, apologies on the lengthy email. Oh and this likely relevant. I'm running FreeBSD 6.3 RELEASE that I installed on July 21. I don't believe any updates have been performed since. I've got a deadlock situation between a fork call on the main thread and another thread. I already know I'm venturing into shady territory when I fork a multithreaded process. However, I don't have a good understanding of exactly what's going on there, which is where I'm hoping you guys can help. Stack dumps are at the bottom of this email. Idea 1: Thread #5 blows up and in its bad state is stuck on a mutex for whatever reason. Thread #1 in the fork call is attempting to reinitialize each mutex in the process space and deadlocks because Thread #5 is hosed and has one of those mutexes. Idea 2: Thread #5 blows up at the same time that Thread #1 makes the fork call. Since at this point in FreeBSD (6.3), the Giant Lock logic is still in there, and these two threads deadlock each other on pthread_mutexattr_init calls. Note that my forked process logic is followed by an execv call, although I just noticed that if execv fails for whatever reason, I'm likely doing some unsafe logging calls before that forked process quits. Any input on whether Idea 1 or Idea 2 are valid? Thank you! - Stephen Here's the main thread (#1) doing the fork call. It's blocked on that pthread_mutexattr_init call. #0 0x28109198 in pthread_sigmask () from /lib/libpthread.so.2 #1 0x28109148 in sigprocmask () from /lib/libpthread.so.2 #2 0x2811360c in pthread_mutexattr_init () from /lib/libpthread.so.2 #3 0x281062db in fork () from /lib/libpthread.so.2 #4 0x2850e04c in blt::Fork () at wrapunix.cpp:978 #5 0x08073e7b in cerebro::ModuleManager::loadModule (this=0x810af40, rModule=@0x816a200) at ModuleManager.cpp:500 #6 0x08073f54 in cerebro::ModuleManager::loadModules (this=0x810af40, mqipctok=@0x80ef134, piddir=@0x80ef124) at ModuleManager.cpp:536 #7 0x08064e26 in cerebro::Cerebro::main (this=0x80ef100, argc=4, argv=0xbfbfe9d4) at Cerebro.cpp:721 #8 0x08069e6a in main (argc=4, argv=0xbfbfe9d4) at Cerebro.cpp:1934 And here's the second (corrupted) thread (#5), that I'm 90% sure is caused by a third-party logging library, unfortunately. #0 0x28114fbf in pthread_mutexattr_init () from /lib/libpthread.so.2 #1 0x28110e29 in _pthread_mutex_trylock () from /lib/libpthread.so.2 #2 0x28112cc6 in pthread_mutex_lock () from /lib/libpthread.so.2 #3 0x2811d3f5 in __error () from /lib/libpthread.so.2 #4 0x28110e29 in _pthread_mutex_trylock () from /lib/libpthread.so.2 #5 0x28112cc6 in pthread_mutex_lock () from /lib/libpthread.so.2 #6 0x2810831a in _spinunlock () from /lib/libpthread.so.2 #7 0x289c5d22 in _UTF8_init () from /lib/libc.so.6 #8 0x28a42580 in _thread_autoinit_dummy_decl_stub () from /lib/libc.so.6 #9 0x0813c280 in ?? () #10 0xbf8fda38 in ?? () #11 0x289c5cc9 in _UTF8_init () from /lib/libc.so.6 #12 0x28a2f593 in sys_nsig () from /lib/libc.so.6 #13 0x2894e5c4 in ?? () from /usr/lib/libstdc++.so.5 #14 0xbf8fd9c8 in ?? () #15 0x2892db3d in operator delete () from /usr/lib/libstdc++.so.5 #16 0x2892db51 in operator delete () from /usr/lib/libstdc++.so.5 #17 0x288ed6fb in std::string::_Rep::_M_destroy () from /usr/lib/libstdc++.so.5 #18 0x28110e29 in _pthread_mutex_trylock () from /lib/libpthread.so.2 #19 0x2811d3f5 in __error () from /lib/libpthread.so.2 #20 0x280fa170 in ?? () #21 0x00000000 in ?? () #22 0xbf8fdb64 in ?? () #23 0x280d46fc in symlook_obj () from /libexec/ld-elf.so.1 From alfred at freebsd.org Tue Oct 28 16:40:26 2008 From: alfred at freebsd.org (Alfred Perlstein) Date: Tue Oct 28 16:40:32 2008 Subject: deadlock in fork call? In-Reply-To: <5B0A3FB3D06E41849325AD3EA98CAD55@OCTOBER> References: <5B0A3FB3D06E41849325AD3EA98CAD55@OCTOBER> Message-ID: <20081028164026.GF83037@elvis.mu.org> I'm pretty sure this is fixed in 6-stable. -Alfred * Stephen Bartnikowski [081028 09:19] wrote: > Hi guys, > > To start, I'm not reporting a bug here. I'm trying to get an idea if I'm on > the right track. Also, apologies on the lengthy email. > > Oh and this likely relevant. I'm running FreeBSD 6.3 RELEASE that I > installed on July 21. I don't believe any updates have been performed since. > > I've got a deadlock situation between a fork call on the main thread and > another thread. I already know I'm venturing into shady territory when I > fork a multithreaded process. However, I don't have a good understanding of > exactly what's going on there, which is where I'm hoping you guys can help. > Stack dumps are at the bottom of this email. > > Idea 1: Thread #5 blows up and in its bad state is stuck on a mutex for > whatever reason. Thread #1 in the fork call is attempting to reinitialize > each mutex in the process space and deadlocks because Thread #5 is hosed and > has one of those mutexes. > > Idea 2: Thread #5 blows up at the same time that Thread #1 makes the fork > call. Since at this point in FreeBSD (6.3), the Giant Lock logic is still in > there, and these two threads deadlock each other on pthread_mutexattr_init > calls. > > Note that my forked process logic is followed by an execv call, although I > just noticed that if execv fails for whatever reason, I'm likely doing some > unsafe logging calls before that forked process quits. > > Any input on whether Idea 1 or Idea 2 are valid? > > Thank you! > > - Stephen > > Here's the main thread (#1) doing the fork call. It's blocked on that > pthread_mutexattr_init call. > > #0 0x28109198 in pthread_sigmask () from /lib/libpthread.so.2 > #1 0x28109148 in sigprocmask () from /lib/libpthread.so.2 > #2 0x2811360c in pthread_mutexattr_init () from /lib/libpthread.so.2 > #3 0x281062db in fork () from /lib/libpthread.so.2 > #4 0x2850e04c in blt::Fork () at wrapunix.cpp:978 > #5 0x08073e7b in cerebro::ModuleManager::loadModule (this=0x810af40, > rModule=@0x816a200) at ModuleManager.cpp:500 > #6 0x08073f54 in cerebro::ModuleManager::loadModules (this=0x810af40, > mqipctok=@0x80ef134, piddir=@0x80ef124) at ModuleManager.cpp:536 > #7 0x08064e26 in cerebro::Cerebro::main (this=0x80ef100, argc=4, > argv=0xbfbfe9d4) at Cerebro.cpp:721 > #8 0x08069e6a in main (argc=4, argv=0xbfbfe9d4) at Cerebro.cpp:1934 > > And here's the second (corrupted) thread (#5), that I'm 90% sure is caused > by a third-party logging library, unfortunately. > > #0 0x28114fbf in pthread_mutexattr_init () from /lib/libpthread.so.2 > #1 0x28110e29 in _pthread_mutex_trylock () from /lib/libpthread.so.2 > #2 0x28112cc6 in pthread_mutex_lock () from /lib/libpthread.so.2 > #3 0x2811d3f5 in __error () from /lib/libpthread.so.2 > #4 0x28110e29 in _pthread_mutex_trylock () from /lib/libpthread.so.2 > #5 0x28112cc6 in pthread_mutex_lock () from /lib/libpthread.so.2 > #6 0x2810831a in _spinunlock () from /lib/libpthread.so.2 > #7 0x289c5d22 in _UTF8_init () from /lib/libc.so.6 > #8 0x28a42580 in _thread_autoinit_dummy_decl_stub () from /lib/libc.so.6 > #9 0x0813c280 in ?? () > #10 0xbf8fda38 in ?? () > #11 0x289c5cc9 in _UTF8_init () from /lib/libc.so.6 > #12 0x28a2f593 in sys_nsig () from /lib/libc.so.6 > #13 0x2894e5c4 in ?? () from /usr/lib/libstdc++.so.5 > #14 0xbf8fd9c8 in ?? () > #15 0x2892db3d in operator delete () from /usr/lib/libstdc++.so.5 > #16 0x2892db51 in operator delete () from /usr/lib/libstdc++.so.5 > #17 0x288ed6fb in std::string::_Rep::_M_destroy () from > /usr/lib/libstdc++.so.5 > #18 0x28110e29 in _pthread_mutex_trylock () from /lib/libpthread.so.2 > #19 0x2811d3f5 in __error () from /lib/libpthread.so.2 > #20 0x280fa170 in ?? () > #21 0x00000000 in ?? () > #22 0xbf8fdb64 in ?? () > #23 0x280d46fc in symlook_obj () from /libexec/ld-elf.so.1 > > _______________________________________________ > freebsd-threads@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-threads > To unsubscribe, send any mail to "freebsd-threads-unsubscribe@freebsd.org" -- - Alfred Perlstein From sbartnikowski at barkinglizards.com Tue Oct 28 17:32:16 2008 From: sbartnikowski at barkinglizards.com (Stephen Bartnikowski) Date: Tue Oct 28 17:32:22 2008 Subject: deadlock in fork call? In-Reply-To: <20081028164026.GF83037@elvis.mu.org> References: <5B0A3FB3D06E41849325AD3EA98CAD55@OCTOBER> <20081028164026.GF83037@elvis.mu.org> Message-ID: <687C5AD700E9484DABB55D5F92A18C10@OCTOBER> Thanks Alfred. I mostly do programming, so I didn't even think to put my admin hat on. Sounds like this patch will do the job. - Stephen > -----Original Message----- > From: Alfred Perlstein [mailto:alfred@freebsd.org] > Sent: Tuesday, October 28, 2008 11:40 AM > To: Stephen Bartnikowski > Cc: freebsd-threads@freebsd.org > Subject: Re: deadlock in fork call? > > I'm pretty sure this is fixed in 6-stable. > > -Alfred > > * Stephen Bartnikowski > [081028 09:19] wrote: > > Hi guys, > > > > To start, I'm not reporting a bug here. I'm trying to get > an idea if > > I'm on the right track. Also, apologies on the lengthy email. > > > > Oh and this likely relevant. I'm running FreeBSD 6.3 RELEASE that I > > installed on July 21. I don't believe any updates have been > performed since. > > > > I've got a deadlock situation between a fork call on the > main thread > > and another thread. I already know I'm venturing into shady > territory > > when I fork a multithreaded process. However, I don't have a good > > understanding of exactly what's going on there, which is > where I'm hoping you guys can help. > > Stack dumps are at the bottom of this email. > > > > Idea 1: Thread #5 blows up and in its bad state is stuck on a mutex > > for whatever reason. Thread #1 in the fork call is attempting to > > reinitialize each mutex in the process space and deadlocks because > > Thread #5 is hosed and has one of those mutexes. > > > > Idea 2: Thread #5 blows up at the same time that Thread #1 > makes the > > fork call. Since at this point in FreeBSD (6.3), the Giant > Lock logic > > is still in there, and these two threads deadlock each other on > > pthread_mutexattr_init calls. > > > > Note that my forked process logic is followed by an execv call, > > although I just noticed that if execv fails for whatever > reason, I'm > > likely doing some unsafe logging calls before that forked > process quits. > > > > Any input on whether Idea 1 or Idea 2 are valid? > > > > Thank you! > > > > - Stephen > > > > Here's the main thread (#1) doing the fork call. It's > blocked on that > > pthread_mutexattr_init call. > > > > #0 0x28109198 in pthread_sigmask () from /lib/libpthread.so.2 > > #1 0x28109148 in sigprocmask () from /lib/libpthread.so.2 > > #2 0x2811360c in pthread_mutexattr_init () from > /lib/libpthread.so.2 > > #3 0x281062db in fork () from /lib/libpthread.so.2 > > #4 0x2850e04c in blt::Fork () at wrapunix.cpp:978 > > #5 0x08073e7b in cerebro::ModuleManager::loadModule > (this=0x810af40, > > rModule=@0x816a200) at ModuleManager.cpp:500 > > #6 0x08073f54 in cerebro::ModuleManager::loadModules > (this=0x810af40, > > mqipctok=@0x80ef134, piddir=@0x80ef124) at ModuleManager.cpp:536 > > #7 0x08064e26 in cerebro::Cerebro::main (this=0x80ef100, argc=4, > > argv=0xbfbfe9d4) at Cerebro.cpp:721 > > #8 0x08069e6a in main (argc=4, argv=0xbfbfe9d4) at Cerebro.cpp:1934 > > > > And here's the second (corrupted) thread (#5), that I'm 90% sure is > > caused by a third-party logging library, unfortunately. > > > > #0 0x28114fbf in pthread_mutexattr_init () from > /lib/libpthread.so.2 > > #1 0x28110e29 in _pthread_mutex_trylock () from > /lib/libpthread.so.2 > > #2 0x28112cc6 in pthread_mutex_lock () from /lib/libpthread.so.2 > > #3 0x2811d3f5 in __error () from /lib/libpthread.so.2 > > #4 0x28110e29 in _pthread_mutex_trylock () from > /lib/libpthread.so.2 > > #5 0x28112cc6 in pthread_mutex_lock () from /lib/libpthread.so.2 > > #6 0x2810831a in _spinunlock () from /lib/libpthread.so.2 > > #7 0x289c5d22 in _UTF8_init () from /lib/libc.so.6 > > #8 0x28a42580 in _thread_autoinit_dummy_decl_stub () from > > /lib/libc.so.6 > > #9 0x0813c280 in ?? () > > #10 0xbf8fda38 in ?? () > > #11 0x289c5cc9 in _UTF8_init () from /lib/libc.so.6 > > #12 0x28a2f593 in sys_nsig () from /lib/libc.so.6 > > #13 0x2894e5c4 in ?? () from /usr/lib/libstdc++.so.5 > > #14 0xbf8fd9c8 in ?? () > > #15 0x2892db3d in operator delete () from /usr/lib/libstdc++.so.5 > > #16 0x2892db51 in operator delete () from /usr/lib/libstdc++.so.5 > > #17 0x288ed6fb in std::string::_Rep::_M_destroy () from > > /usr/lib/libstdc++.so.5 > > #18 0x28110e29 in _pthread_mutex_trylock () from > /lib/libpthread.so.2 > > #19 0x2811d3f5 in __error () from /lib/libpthread.so.2 #20 > 0x280fa170 > > in ?? () > > #21 0x00000000 in ?? () > > #22 0xbf8fdb64 in ?? () > > #23 0x280d46fc in symlook_obj () from /libexec/ld-elf.so.1 > > > > _______________________________________________ > > freebsd-threads@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-threads > > To unsubscribe, send any mail to > "freebsd-threads-unsubscribe@freebsd.org" > > -- > - Alfred Perlstein > From info at yourkit.com Thu Oct 30 10:41:18 2008 From: info at yourkit.com (Vladimir Kondratyev) Date: Thu Oct 30 10:41:25 2008 Subject: getting CPU time for particulat thread Message-ID: <678291552.20081030201553@yourkit.com> Hi folks, is there any ability to get CPU time consumed by particular thread? We are porting YourKit Java Profiler (http://www.yourkit.com) to FreeBSD platform. Everything works almost fine, but we cannot find a way how to receive thread CPU times? Any help is greatly appreciated. Thank you in advance. Sincerely, Vladimir Kondratyev YourKit, LLC http://www.yourkit.com "Don't get lost in data, get information!" From davidxu at freebsd.org Thu Oct 30 18:20:38 2008 From: davidxu at freebsd.org (David Xu) Date: Thu Oct 30 18:20:44 2008 Subject: getting CPU time for particulat thread In-Reply-To: <678291552.20081030201553@yourkit.com> References: <678291552.20081030201553@yourkit.com> Message-ID: <490A5DD2.8030703@freebsd.org> Vladimir Kondratyev wrote: > Hi folks, > > is there any ability to get CPU time consumed by particular thread? > > We are porting YourKit Java Profiler (http://www.yourkit.com) to FreeBSD > platform. Everything works almost fine, but we cannot find a way how to receive > thread CPU times? > > Any help is greatly appreciated. > > Thank you in advance. > > Sincerely, > Vladimir Kondratyev > YourKit, LLC > http://www.yourkit.com > "Don't get lost in data, get information!" > if you just want to get current thread's time, there is a feature in -HEAD branch, call clock_gettime(CLOCK_THREAD_CPUTIME_ID, &ts) to get total cpu time for the current thread. Regards, David Xu