PS3 livelock and pmap_remove()

[Nathan Whitehorn]

Now that my SLB allocation issue is solved, with help with Matthew and
Alan, I have another VM puzzler.

I have a simple program that tries to use all the memory on the system,
which isn't very much on the PS3, so I use it to test swap as well.
Shortly after it begins paging, the system locks up completely. I
managed to duplicate this behavior on an emulator, and found out what it
is actually doing. Somehow pmap_remove() is being called with arguments
pmap_remove(userpmap, PAGE_SIZE, VM_MAXUSER_ADDRESS = USRSTACK). For
powerpc64, VM_MAXUSER_ADDRESS is 0x7ffffffffffff000, so there are 10^15
pages to unmap in that range and it was busy taking until the end of
time unmapping them all.

Here's the trace from KDB:

moea64_remove()
pmap_remove()
vm_daemon()
fork_exit()
fork_trampoline()
----end-----

Does anyone have any idea why this is happening?

Thanks,
-Nathan