xenstore memory issue
Andrew Thompson
thompsa at FreeBSD.org
Tue Feb 10 03:25:33 UTC 2015
On 9 February 2015 at 23:28, Roger Pau Monné <roger.pau at citrix.com> wrote:
> Hello,
>
> El 09/02/15 a les 10.39, Andrew Thompson ha escrit:
> > Hi,
> >
> >
> > I have three VMs with Rackspace and one is behaving oddly with xenstore
> > memory consumption. Here are the kernel versions and vmstat -m results.
> >
>
> As you can see the third VM is using 100MB in xenstore memory and it seems
> > to be climbing by 1-2MB per hour. Eventually all the processes go in to
> > pfault state and it grinds to a halt.
>
> That's certainly weird, are you doing something different on this VM as
> compared to the others? Did you hot-add a nic, disk or ballooned memory?
>
> Has the VM been saved/restored or migrated?
>
> Tracking down this kind of xenstore leaks can be difficult without
> having a way to reproduce them.
>
>
A bit of trial and error with dtrace has narrowed this down. I can cause
the leak by just opening /dev/xen/xenstore
int main() {
open("/dev/xen/xenstore", O_RDWR, 0);
}
# vmstat -m | grep xenstore; ./open; vmstat -m | grep xenstore
xenstore 8739 104797K - 56078 16,32,64,128,256,512
xenstore 8740 104809K - 56079 16,32,64,128,256,512
Using dtrace probes I can see that xs_dev_close is never called.
# dtrace -n 'fbt::xs_dev_open:{} fbt::xs_dev_close: {} dtmalloc::xenstore:
{}'
# ./open
CPU FUNCTION
0 -> xs_dev_open
0 | xenstore:malloc
0 <- xs_dev_open
This is on 10.0-RELEASE-p12. I get the same result with
`/usr/local/bin/xenstore-read
domid` but the above c program is enough too.
regards,
Andrew
More information about the freebsd-xen
mailing list