System becomes unresponsive when it runs out of memory

[Yuri]

I observed this several times: when the system with 24GB RAM +4GB swap 
runs out of memory and swap, it loses the ability to be connected to. 
ssh connects, but the second factor authentication 
(google-authenticator) fails to produce the code. This lasts for an 
extended period of time (~15 min), after which ssh connection eventually 
succeeds.


My question is: isn't it supposed to kill the largest process when it 
runs out of memory? IMO, this is a bug that system doesn't handle 
memory-full condition well and hangs up like this.


Another bothersome thing is that I can't really account for the consumed 
memory. There are several large processes with sizes ~50GB, but 
madvise(MADV_FREE) has been called on most of that memory by the 
perftools malloc library. So I don't understand why this memory-full 
occurred in the first place. Is it possible that madvise(MADV_FREE) 
doesn't discard memory?


Yuri