FreeBSD 9.1 excessive memory allocations [SOLVED]

[Ronald Klop]

On Wed, 27 Mar 2013 19:33:46 +0100, Unga <unga888 at yahoo.com> wrote:

>
>
> ----- Original Message -----
>
>> From: Ian Lepore <ian at FreeBSD.org>
>> To: Unga <unga888 at yahoo.com>
>> Cc: "freebsd-stable at freebsd.org" <freebsd-stable at FreeBSD.org>
>> Sent: Wednesday, March 27, 2013 2:06 PM
>> Subject: Re: FreeBSD 9.1 excessive memory allocations
>>
>> On Tue, 2013-03-26 at 11:35 -0700, Unga wrote:
>>>  Hi all
>>>
>>>
>>>  I have a heavily threaded C application, developed on an Intel Core i5
>> laptop (2 cores) running FreeBSD 8.1-RELEASE.
>>>
>>>  When this application compile and run on another Intel Core i7 laptop  
>>> (4
>> cores) running FreeBSD 9.1-RELEASE, this application immediately starts  
>> grabbing
>> memory by over 100MB per second and soon exit with not enough RAM.
>>>
>>>
>>>  Both laptops having 4GB RAM.
>>>
>>>  All malloc and free are mutex locked.
>>>
>>>  Very rarely this problem happens on the i5 (2 cores) laptop too, but  
>>> on the
>> i7 laptop, it happens every time.
>>>
>>>  Appreciate any feedback to identify and fix this issue.
>>>
>>>  Best regards
>>>  Unga
>>>
>>
>> Too many moving parts, you need to partition the problem.  Is it the
>> change in OS (and especially libraries) that causes the problem, or the
>> change in the number of cores (more concurrent threads) is exposing some
>> sort of application-side race condition?  Given the fact that it does
>> occur on 2 cores + freebsd 8.1, even if more rarely, it's almost surely
>> an application problem.
>> Perhaps you could use a tool such as valgrind to help track down the
>> runaway allocations?
>>
>> Another way to expose thread race problems is to force more thread
>> context switches.  A blunt tool for doing so is to set hz=5000 or even
>> higher in /boot/loader.conf.  I've never done that on a desktop or
>> laptop system, only on embedded systems, but it should still work okay.
>> If changing the application code is easier, you can get a similar effect
>> by creating a thread whose only job is to preempt other threads, by
>> using rtprio_thread() to set it to real time priority, then just have it
>> sleep for short random intervals (microseconds), all it does is wakes up
>> and goes right back to sleep.
>>
>> Also, FYI, there's no need to use a mutex in your application to protect
>> allocations.  The memory allocator in libc is thread-safe, and in fact
>> is particularly designed for efficient multi-threaded allocation.
>>
>> -- Ian
>>
>
> Dear Tony, Alexander, Jan, Ian and Jeremy
>
> Thank you very much for your very valuable comments.
>
> Problem seems to be solved. But require more testing.
>
> 1. Fixed an application-level crucial bug. This is nearly a 7000 lines C  
> app. It was really hard to see as the application is designed with 8  
> dedicated threads.
>
> 2. At run-time, this application shoots up to about 400 dynamic threads  
> on the i7 machine, whereas on the i5 machine, it only shoots up 57  
> dynamic threads. It was bit scaring, therefore, made a hard limit on  
> total number of threads to 64.
>
> Regarding mutex locks on allocations, as per the malloc(3), it says  
> small and medium size allocations are done from per thread caches,  
> therefore, thread-safe. My allocations are large in nature, about 5-7MB.

The per thread caches are for speed. Not for thread-safety.

Ronald.