xenstore memory issue

[Roger Pau Monné]

Hello,

El 09/02/15 a les 10.39, Andrew Thompson ha escrit:
> Hi,
> 
> 
> I have three VMs with Rackspace and one is behaving oddly with xenstore
> memory consumption. Here are the kernel versions and vmstat -m results.
> 
> FreeBSD us.e.com 10.0-RELEASE-p9 FreeBSD 10.0-RELEASE-p9 #0: Mon Sep 15
> 14:35:52 UTC 2014
> root at amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC
>  amd64
> 
>        xenbus    16     2K       -       86  16,32,64,256
>      xenstore   409  4837K       - 38424052  16,32,64,128,256
>       xen_hvm     2     8K       -        2  4096
>      xen_intr    25     4K       -       25  128
> 
> 
> FreeBSD uk.e.com 10.0-RELEASE-p12 FreeBSD 10.0-RELEASE-p12 #0: Tue Nov  4
> 05:07:17 UTC 2014
> root at amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC
>  amd64
> 
>        xenbus    11     2K       -       83  16,32,64,256
>      xenstore   198  2317K       - 43428137  16,32,64,128,256,512
>       xen_hvm     2     8K       -        2  4096
>      xen_intr    24     3K       -       24  128
> 
> 
> FreeBSD au.e.com 10.0-RELEASE-p12 FreeBSD 10.0-RELEASE-p12 #0: Tue Nov  4
> 05:07:17 UTC 2014
> root at amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC
>  amd64
> 
>        xenbus    11     2K       -       83  16,32,64,256
>      xenstore  8477 101653K       -    55249  16,32,64,128,256,512
>       xen_hvm     2     8K       -        2  4096
>      xen_intr    14     2K       -       14  128
> 
> 
> As you can see the third VM is using 100MB in xenstore memory and it seems
> to be climbing by 1-2MB per hour. Eventually all the processes go in to
> pfault state and it grinds to a halt.

That's certainly weird, are you doing something different on this VM as
compared to the others? Did you hot-add a nic, disk or ballooned memory?

Has the VM been saved/restored or migrated?

Tracking down this kind of xenstore leaks can be difficult without
having a way to reproduce them.

> 
> How should I be debugging this? Is it either a local leak or the Xen host
> is to blame?

Even if the host is doing something weird we should be able to cope with
it, or at least detect it.

Roger.