pthread programming eats up resources (My or FreeBSD's fault?)

[Andre Albsmeier]

On Wed, 19-Feb-2014 at 23:08:24 +0800, Erich Dollansky wrote:
> Hi,
> 
> as I understand your program, it creates 1000 threads, waits a but and
> then starts again creating 1000 threads until something kill it. 

Right. 1000 threads are started, do nothing and thus terminate
immediately. The value of 25 ms for usleep is just a good one
for my machine to demonstrate the problem quickly while being
sure that all 1000 threads have gone (verified with ps -H).
You can chose a bigger value here and it will just take more
time...

> 
> From my point of view, your program depends very much on the default
> settings of the environment. If the environment allows the immediate
> execution of the new thread, you will never get many threads.
> Otherwise, the number of threads hanging around could add up. It also
> depends on the number of CPUs/cores your system has.

But I never have more than 1000 threads. This is why I made pth2 where
I explicitly wait for all 1000 threads to terminate before firing up
the next bunch.

So for me it seems as if every (already died away) thread leaves some
memory behind which does not get free'ed or whatever. But it was my
understanding that pthread_detach() or pthread_join() would clean
all this up after a thread has finished.

> 
> But your are right, it should not crash on a modern machine but it
> still could use some amount of memory.

But should it still grow with every bunch of 1000 threads being
started while the previous 1000 are all gone already? I'd agree
if the behaviour was like on the Linux machine where it seems
that some kind of garbage collection kicks in for VIRT after a
while but RES never goes above 1MB there. On FreeBSD it even
makes the box swap...

	-Andre

> 
> Erich
> 
> On Wed,
> 19 Feb 2014 15:47:28 +0100 Andre Albsmeier <mail at ma17.ata.myota.org>
> wrote:
> 
> > [Commenting my own mail from below and CC'ing freebsd-threads.]
> > 
> > I have tested my code below on a Linux box (3.2.0) and here the
> > behaviour is different and seems correct: While watching with
> > top(1), VIRT climbs up to a few GB and collapses then to a few
> > MB (somehow reminding me of some kind of garbage collection).
> > Important thing is that RES always stays below 1MB.
> > 
> > On FreeBSD SIZE and RES are about 100MB apart but both are
> > increasing util 2GB and pth1 dies with 
> > 
> > Cannot map anonymous memory
> > Out of memory
> > 
> > So the question is: Is my programme buggy and Linux works around
> > this bug or is there some kind of memory leak in the pthreads
> > code in FreeBSD?
> > 
> > 	-Andre
> > 
> > ----- Forwarded message  -----
> > 
> > Well, as these are my first steps regarding thread programming,
> > it's probably my fault...
> > 
> > Why does this programme slowly grow and grow until it hits
> > resource limits?
> > 
> > ----- snip pth1.c -----
> > 
> > #include <stdio.h>
> > #include <unistd.h>
> > #include <pthread.h>
> > 
> > void* mythread( void* arg )
> > {
> >   return NULL;
> > }
> > 
> > int main( int argc, const char* const argv[] )
> > {
> >   pthread_t pthr;
> >   int i;
> > 
> >   while( 1 ) {
> > 
> >     for( i=1000; i; i-- )
> >       if( pthread_create( &pthr, NULL, mythread, NULL ) != 0 )
> >         fprintf( stderr, "pthread_create\n" );
> >       else
> >         pthread_detach( pthr );
> > 
> >     putchar( '.' );
> >     fflush( stdout );
> >     usleep( 25000 );
> >   }
> > }
> > 
> > ----- snap -----
> > 
> > Just to be sure I have also created the non-detaching version
> > which behaves in the same way:
> > 
> > ----- snip pth2.c -----
> > 
> > #include <stdio.h>
> > #include <unistd.h>
> > #include <pthread.h>
> > 
> > #define M 1000
> > 
> > pthread_t pthr[M];
> > 
> > void* mythread( void* arg )
> > {
> >   return NULL;
> > }
> > 
> > int main( int argc, const char* const argv[] )
> > {
> >   int i;
> > 
> >   while( 1 ) {
> > 
> >     for( i=M; i; i-- )
> >       if( pthread_create( &pthr[i], NULL, mythread, NULL ) != 0 )
> >         fprintf( stderr, "pthread_create\n" );
> > 
> >     for( i=M; i; i-- )
> >       if( pthread_join( pthr[i], NULL ) != 0 )
> >         fprintf( stderr, "pthread_join\n" );
> > 
> >     putchar( '.' );
> >     fflush( stdout );
> >     usleep( 25000 );
> >   }
> > }
> > 
> > ----- snap -----
> > 
> > Compile them using -pthread and watch their ps output in another
> > window (FreeBSD-9.2 but that shouldn't matter).
> > 
> > So what am I doing wrong here?
> > 
> > Thanks,
> > 
> > 	-Andre
> > _______________________________________________
> > freebsd-threads at freebsd.org mailing list
> > http://lists.freebsd.org/mailman/listinfo/freebsd-threads
> > To unsubscribe, send any mail to
> > "freebsd-threads-unsubscribe at freebsd.org"
> 
> 

-- 
echo "cdec cdec efg~ efg~ L8gagfL4ec L8gagfL4ec c<g>c~ c<g>c~" >/dev/speaker