bhyve uses all available memory during IO-intensive operations

[Marcelo Araujo]

2017-12-01 17:53 GMT+08:00 Shane Ambler <FreeBSD at shaneware.biz>:

> On 01/12/2017 13:43, Allan Jude wrote:
> > On 2017-11-30 22:10, Dustin Wenz wrote:
> >> I am using a zvol as the storage for the VM, and I do not have any ARC
> >> limits set. However, the bhyve process itself ends up grabbing the vast
> >> majority of memory.Â
> >>
> >> I’ll run a test tomorrow to get the exact output from top.
> >>
> >> Â  Â - .Dustin
> >>
> >> On Nov 30, 2017, at 5:28 PM, Allan Jude <allanjude at freebsd.org
> >> <mailto:allanjude at freebsd.org>> wrote:
> >>
> >>> On 11/30/2017 18:15, Dustin Wenz wrote:
> >>>> I'm using chyves on FreeBSD 11.1 RELEASE to manage a few VMs (guest
> >>>> OS is also FreeBSD 11.1). Their sole purpose is to house some
> >>>> medium-sized Postgres databases (100-200GB). The host system has 64GB
> >>>> of real memory and 112GB of swap. I have configured each guest to
> >>>> only use 16GB of memory, yet while doing my initial database imports
> >>>> in the VMs, bhyve will quickly grow to use all available system
> >>>> memory and then be killed by the kernel:
> >>>>
> >>>> Â  Â kernel: swap_pager: I/O error - pageout failed; blkno 1735,size
> >>>> 4096, error 12
> >>>> Â  Â kernel: swap_pager: I/O error - pageout failed; blkno 1610,size
> >>>> 4096, error 12
> >>>> Â  Â kernel: swap_pager: I/O error - pageout failed; blkno 1763,size
> >>>> 4096, error 12
> >>>> Â  Â kernel: pid 41123 (bhyve), uid 0, was killed: out of swap space
>
> That's the type of errors I see when wired jumps high. I'm not seeing
> this from bhyve but when your watching top, keep an eye on the wired
> amount.
>
> --
> FreeBSD - the place to B...Sharing Devices
>
> Shane Ambler
>
> _______________________________________________
> freebsd-virtualization at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> To unsubscribe, send any mail to "freebsd-virtualization-
> unsubscribe at freebsd.org"
>

As Allan said, it is likely to be your ARC cache holding memory and
unfortunately right now all these swap out doesn't play well with the
combination of zfs + bhyve.

Try to tune your vfs.zfs.arc_max to a minimum where you give memory space
enough to your VM.

What I'm doing now is, launch a VM get the amount of memory and remove it
from vfs.zfs.arc_max, as soon as the VM stops, I give the memory back to
vfs.zfs.arc_max.

Best,
-- 

-- 
Marcelo Araujo            (__)araujo at FreeBSD.org
\\\'',)http://www.FreeBSD.org <http://www.freebsd.org/>   \/  \ ^
Power To Server.         .\. /_)