Strange Swapping Issues(?)

[Jeremy Chadwick]

On Wed, Apr 14, 2010 at 06:44:58AM +0200, Gabor PALI wrote:
> Hello there,
> 
> I am running a FreeBSD/amd64 8-STABLE with GENERIC kernel as of
> February 17 on my box (quad core, 2 GB RAM), and recently I spot some
> interesting problems in my logs.  My machine runs two instances of a
> client in two separate chroot environments in parallel with 32-bit and
> 64-bit userlands respectively, doing a nightly building and testing.
> According to the logs it puts a nice load on my system, though it
> still seems to be working fine.
> 
> Except one thing: it produces strange error messages on swap space
> without an apparent reason (at least to me):
> 
> xxx# tail -f /var/log/messages
> Apr 14 05:26:44 xxx kernel: swap_pager_getswapspace(4): failed
> Apr 14 05:26:44 xxx kernel: swap_pager_getswapspace(16): failed
> Apr 14 05:26:44 xxx kernel:
> Apr 14 05:26:44 xxx kernel: swap_pager_getswapspace(3): failed
> Apr 14 05:26:44 xxx kernel: swap_pager_getswapspace(16): failed
> Apr 14 05:26:44 xxx kernel: swap_pager_getswapspace(3): failed
> Apr 14 05:26:45 xxx kernel: swap_pager_getswapspace(16): failed
> Apr 14 05:26:45 xxx kernel: swap_pager_getswapspace(12): failed
> Apr 14 05:26:45 xxx kernel: swap_pager_getswapspace(2): failed
> Apr 14 05:26:45 xxx kernel: swap_pager_getswapspace(16): failed
> Apr 14 05:26:45 xxx kernel: swap_pager_getswapspace(16): failed
> Apr 14 05:26:45 xxx kernel: pid 7388 (throwto003), uid 1001, was
> killed: out of swap space
> Apr 14 05:26:45 xxx kernel: swap_pager_getswapspace(16): failed
> Apr 14 05:26:45 xxx kernel: swap_pager_getswapspace(8): failed
> Apr 14 05:26:45 xxx kernel: swap_pager_getswapspace(16): failed
> Apr 14 05:26:45 xxx kernel: swap_pager_getswapspace(16): failed
> Apr 14 05:26:45 xxx kernel: swap_pager_getswapspace(8): failed
> Apr 14 05:26:45 xxx kernel: swap_pager_getswapspace(16): failed
> Apr 14 05:26:45 xxx kernel: swap_pager_getswapspace(12): failed
> Apr 14 05:26:45 xxx kernel: swap_pager_getswapspace(9): failed
> Apr 14 05:26:45 xxx kernel: swap_pager_getswapspace(16): failed
> Apr 14 05:26:45 xxx kernel: swap_pager_getswapspace(9): failed
> Apr 14 05:26:45 xxx kernel: swap_pager_getswapspace(3): failed
> Apr 14 05:26:45 xxx kernel: swap_pager_getswapspace(16): failed
> Apr 14 05:26:45 xxx kernel: swap_pager_getswapspace(9): failed
> Apr 14 05:26:45 xxx kernel: swap_pager_getswapspace(3): failed
> ^C
> xxx# swapinfo -h
> Device          1K-blocks     Used    Avail Capacity
> /dev/ad0s1b       4194304     112M     3.9G     3%

The swapinfo command you ran was not run at 05:26 in the morning.  You
should probably set up a small script, run via cronjob, that logs
swapinfo -h output to a file somewhere (rotate it if you want via
newsyslog.conf).

You may have something running on the system that spirals out of
control, such as a web board script being pounded to death, or something
that's forking excessively.

I'd also recommend having the script output "top -b -o res 100", which
will give you the top 100 processes on the machine sorted by RSS
(non-shared) memory usage.  I don't know of a way to show "the amount of
swap used by process N, or all processes", since it's transparently
handled by the VM.  So I'm making the assumption RSS will be large.

-- 
| Jeremy Chadwick                                   jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.              PGP: 4BD6C0CB |