process eating up all memory - what should happen next?

[Anton Shterenlikht]

I have a process that eats up al memory,
in my case science/paraview if I try to
analyse a large model. What should FreeBSD
do when a process tries to use all RAM or more?

I my case I get a complete freeze, can't even
login from the console, and requiring a cold
reboot. I guess this is not supposed to happen,
but what is supposed to happen in situations like this?

This is on ia64, so it might be something to
do with instability there.

Thanks

Anton