powerpc64 malloc limit?

Wed Nov 30 21:24:58 UTC 2011

On Wed, Nov 30, 2011 at 09:58:19PM +0100, Andreas Tobler wrote:
> On 30.11.11 21:01, Kostik Belousov wrote:
> >On Wed, Nov 30, 2011 at 06:44:21PM +0100, Andreas Tobler wrote:
> >>On 30.11.11 18:09, Kostik Belousov wrote:
> >>>On Wed, Nov 30, 2011 at 05:53:04PM +0100, Andreas Tobler wrote:
> >>>>On 30.11.11 17:22, Kostik Belousov wrote:
> >>>>>On Wed, Nov 30, 2011 at 06:24:41AM +0100, Andreas Tobler wrote:
> >>>>>>All,
> >>>>>>
> >>>>>>while working on gcc I found a very strange situation which renders my
> >>>>>>powerpc64 machine unusable.
> >>>>>>The test case below tries to allocate that much memory as 'wanted'. 
> >>>>>>The
> >>>>>>same test case on amd64 returns w/o trying to allocate mem because the
> >>>>>>size is far to big.
> >>>>>>
> >>>>>>I couldn't find the reason so far, that's why I'm here.
> >>>>>>
> >>>>>>As Nathan pointed out the VM_MAXUSER_SIZE is the biggest on powerpc64:
> >>>>>>#define VM_MAXUSER_ADDRESS      (0x7ffffffffffff000UL)
> >>>>>>
> >>>>>>So, I'd expect a system to return an allocation error when a user 
> >>>>>>tries
> >>>>>>to allocate too much memory and not really trying it and going to be
> >>>>>>unusable. Iow, I'd exepect the situation on powerpc64 as I see on 
> >>>>>>amd64.
> >>>>>>
> >>>>>>Can anybody explain me the situation, why do I not have a working 
> >>>>>>limit
> >>>>>>on powerpc64?
> >>>>>>
> >>>>>>The machine itself has 7GB RAM and 12GB swap. The amd64 where I 
> >>>>>>compared
> >>>>>>has around 4GB/4GB RAM/swap.
> >>>>>>
> >>>>>>TIA,
> >>>>>>Andreas
> >>>>>>
> >>>>>>include<stdlib.h>
> >>>>>>#include<stdio.h>
> >>>>>>
> >>>>>>int main()
> >>>>>>{
> >>>>>>          void *p;
> >>>>>>
> >>>>>>          p = (void*) malloc (1152921504606846968ULL);
> >>>>>>          if (p != NULL)
> >>>>>>                  printf("p = %p\n", p);
> >>>>>>
> >>>>>>          printf("p = %p\n", p);
> >>>>>>          return (0);
> >>>>>>}
> >>>>>
> >>>>>First, you should provide details of what consistutes 'the unusable
> >>>>>machine situation' on powerpc.
> >>>>
> >>>>I can not login anymore, everything is stuck except the core control
> >>>>mechanisms for example the fan controller.
> >>>>
> >>>>Top reports 'ugly' figures, below from a earlier try:
> >>>>
> >>>>last pid:  6790;  load averages:  0.78,  0.84,  0.86    up 0+00:34:52
> >>>>22:42:29 47 processes:  1 running, 46 sleeping
> >>>>CPU:  0.0% user,  0.0% nice, 15.4% system, 11.8% interrupt, 72.8% idle
> >>>>Mem: 5912M Active, 570M Inact, 280M Wired, 26M Cache, 104M Buf, 352K 
> >>>>Free
> >>>>Swap: 12G Total, 9904M Used, 2383M Free, 80% Inuse, 178M Out
> >>>>
> >>>>    PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU
> >>>>COMMAND
> >>>>   6768 andreast      1  52    01073741824G  6479M pfault  1   0:58
> >>>>18.90% 31370.
> >>>>
> >>>>And after my mem and swap are full I see swap_pager_getswapspace(16)
> >>>>failed.
> >>>>
> >>>>In this state I can only power-cycle the machine.
> >>>>
> >>>>>That said, on amd64 the user map is between 0 and 0x7fffffffffff, which
> >>>>>obviously less then the requested allocation size 0x100000000000000.
> >>>>>If you look at the kdump output on amd64, you will see that malloc()
> >>>>>tries to mmap() the area, fails and retries with obreak(). Default
> >>>>>virtual memory limit is unlimited, so my best quess is that on amd64
> >>>>>vm_map_findspace() returns immediately.
> >>>>>
> >>>>>On powerpc64, I see no reason why vm_map_entry cannot be allocated, but
> >>>>>please note that vm object and pages shall be only allocated on demand.
> >>>>>So I am curious how does your machine breaks and where.
> >>>>
> >>>>I would expect that the 'system' does not allow me to allocate that much
> >>>>of ram.
> >>>
> >>>Does the issue with machine going into limbo reproducable with the code
> >>>you posted ?
> >>
> >>If I understand you correctly, yes. I can launch the test case and the
> >>machine is immediately unusable. Means I can not kill the process nor
> >>can I log in. Also, top does not show anything useful.
> >Again, let me restate my question: the single mmap() of the huge size is
> >enough for powerpc64 machine to break apart ?
> 
> I can't answer. I don't know yet.
> 
> >What happen if you insert sleep(1000000); call before return ? Do not kill
> >the process, I want to know is machine dead while the process sleeps.
> 
> Ok, during the 'sleep' the machine is usable. top is reporting figures, 
> I can log in and edit files. The process runs now for aboutt 30'.
> 
> When I kill the process, I do not get back to the shell nor can I log 
> in. Also top stops reporting.
> But as you said, I didn't kill in this run.
Then, as Alan Cox pointed out, caused by the approach taken in powerpc64
pmap to handle pmap_remove(). It is definitely arch-specific.

> 
> Thanks,
> Andreas
> 
> [bohrium:~] andreast% ./mega_malloc
> p = 0x50400000
> p = 0x50400000
> ...
> 
> last pid:  1279;  load averages:  0.05,  0.03,  0.04    up 0+00:30:34 
> 21:56:40
> 41 processes:  1 running, 40 sleeping
> CPU:  0.0% user,  0.0% nice,  1.2% system,  0.2% interrupt, 98.6% idle
> Mem: 28M Active, 26M Inact, 89M Wired, 8K Cache, 725M Buf, 6642M Free
> Swap: 12G Total, 12G Free
> 
>   PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU 
> COMMAND
>  1219 andreast      1  24    01073741824G  1608K nanslp  0   0:00 
> 0.00% mega_m
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 196 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-arch/attachments/20111130/c0a9f0fc/attachment.pgp