powerpc64 malloc limit?

Wed Nov 30 18:12:21 UTC 2011

On Wed, Nov 30, 2011 at 9:44 AM, Andreas Tobler <andreast-list at fgznet.ch> wrote:
> On 30.11.11 18:09, Kostik Belousov wrote:
>>
>> On Wed, Nov 30, 2011 at 05:53:04PM +0100, Andreas Tobler wrote:
>>>
>>> On 30.11.11 17:22, Kostik Belousov wrote:
>>>>
>>>> On Wed, Nov 30, 2011 at 06:24:41AM +0100, Andreas Tobler wrote:
>>>>>
>>>>> All,
>>>>>
>>>>> while working on gcc I found a very strange situation which renders my
>>>>> powerpc64 machine unusable.
>>>>> The test case below tries to allocate that much memory as 'wanted'. The
>>>>> same test case on amd64 returns w/o trying to allocate mem because the
>>>>> size is far to big.
>>>>>
>>>>> I couldn't find the reason so far, that's why I'm here.
>>>>>
>>>>> As Nathan pointed out the VM_MAXUSER_SIZE is the biggest on powerpc64:
>>>>> #define VM_MAXUSER_ADDRESS      (0x7ffffffffffff000UL)
>>>>>
>>>>> So, I'd expect a system to return an allocation error when a user tries
>>>>> to allocate too much memory and not really trying it and going to be
>>>>> unusable. Iow, I'd exepect the situation on powerpc64 as I see on
>>>>> amd64.
>>>>>
>>>>> Can anybody explain me the situation, why do I not have a working limit
>>>>> on powerpc64?
>>>>>
>>>>> The machine itself has 7GB RAM and 12GB swap. The amd64 where I
>>>>> compared
>>>>> has around 4GB/4GB RAM/swap.
>>>>>
>>>>> TIA,
>>>>> Andreas
>>>>>
>>>>> include<stdlib.h>
>>>>> #include<stdio.h>
>>>>>
>>>>> int main()
>>>>> {
>>>>>          void *p;
>>>>>
>>>>>          p = (void*) malloc (1152921504606846968ULL);
>>>>>          if (p != NULL)
>>>>>                  printf("p = %p\n", p);
>>>>>
>>>>>          printf("p = %p\n", p);
>>>>>          return (0);
>>>>> }
>>>>
>>>> First, you should provide details of what consistutes 'the unusable
>>>> machine situation' on powerpc.
>>>
>>> I can not login anymore, everything is stuck except the core control
>>> mechanisms for example the fan controller.
>>>
>>> Top reports 'ugly' figures, below from a earlier try:
>>>
>>> last pid:  6790;  load averages:  0.78,  0.84,  0.86    up 0+00:34:52
>>> 22:42:29 47 processes:  1 running, 46 sleeping
>>> CPU:  0.0% user,  0.0% nice, 15.4% system, 11.8% interrupt, 72.8% idle
>>> Mem: 5912M Active, 570M Inact, 280M Wired, 26M Cache, 104M Buf, 352K Free
>>> Swap: 12G Total, 9904M Used, 2383M Free, 80% Inuse, 178M Out
>>>
>>>    PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU
>>> COMMAND
>>>   6768 andreast      1  52    01073741824G  6479M pfault  1   0:58
>>> 18.90% 31370.
>>>
>>> And after my mem and swap are full I see swap_pager_getswapspace(16)
>>> failed.
>>>
>>> In this state I can only power-cycle the machine.
>>>
>>>> That said, on amd64 the user map is between 0 and 0x7fffffffffff, which
>>>> obviously less then the requested allocation size 0x100000000000000.
>>>> If you look at the kdump output on amd64, you will see that malloc()
>>>> tries to mmap() the area, fails and retries with obreak(). Default
>>>> virtual memory limit is unlimited, so my best quess is that on amd64
>>>> vm_map_findspace() returns immediately.
>>>>
>>>> On powerpc64, I see no reason why vm_map_entry cannot be allocated, but
>>>> please note that vm object and pages shall be only allocated on demand.
>>>> So I am curious how does your machine breaks and where.
>>>
>>> I would expect that the 'system' does not allow me to allocate that much
>>> of ram.
>>
>> Does the issue with machine going into limbo reproducable with the code
>> you posted ?
>
> If I understand you correctly, yes. I can launch the test case and the
> machine is immediately unusable. Means I can not kill the process nor can I
> log in. Also, top does not show anything useful.
>
> The original test case where I discovered this behavior behaves a bit
> different.
> http://gcc.gnu.org/viewcvs/trunk/libstdc%2B%2B-v3/testsuite/23_containers/vector/bool/modifiers/insert/31370.cc?revision=169421&view=markup
>
> Here I can follow how the ram and swap is eaten up. Top is reporting the
> figures. If everything is 'full', the swaper errors start to appear on the
> console.
>
>> Or, do you need to actually touch the pages in the allocated region ?
>
> If I have to, how would I do that?
>
>> If the later (and I do expect that), then how many pages do you need
>> to touch before machine breaks ? Is it single access that causes the
>> havoc, or you need to touch the amount approximately equal to RAM+swap ?
>
> Andreas

ia64 had some vaguely related excitement earlier in its life.    If
you created a 1TB sparse file and mmap'ed it over and over, tens,
maybe hundreds of thousands of times, certain VM internal state got
way out of hand.  mmaping was fine, but unmapping took 36 hours of cpu
runtime when I killed the process.  It got so far out of hand because
of the way ia64 handled just-in-time mappings on vhpt misses.

It sounds a lot to me like ppc64 vm system consumes system resources
to map something, which is unbounded, and those large address space
operations are causing most/all of system ram to be wired for VM
management and leading to a lockup.

AMD64's user address space isn't actually all that big compared to
other 64 bit systems.

-- 
Peter Wemm - peter at wemm.org; peter at FreeBSD.org; peter at yahoo-inc.com; KI6FJV
"All of this is for nothing if we don't go to the stars" - JMS/B5
"If Java had true garbage collection, most programs would delete
themselves upon execution." -- Robert Sewell