Options for zfs inside a VM backed by zfs on the host

[Marie]

On Thu, Aug 27, 2015 at 11:42 AM Matt Churchyard via freebsd-virtualization
<freebsd-virtualization at freebsd.org> wrote:

> > On Wed, Aug 26, 2015 at 11:10:44PM -0700, Marcus Reid wrote:
> > On Wed, Aug 26, 2015 at 05:25:52PM -0400, Vick Khera wrote:
> > > > Opinions? Preferably well-reasoned ones. :)
> > > >
> > > However, having the ARC eating up lots of memory twice seems pretty
> > > bletcherous.  You can probably do some tuning to reduce that, but I
> > > never liked tuning the ARC much.
>
> > I just realized that you can turn primarycache off per-dataset.  Does it
> make more sense to turn primarycache=none on the zvol on the host, or > on
> the datasets in the vm?  I'm thinking on the host, but it might be worth
> experimenting.
>
> I'd be very wary of disabling ARC on the main host, it can have pretty
> serious side effects. It could possibly be useful in the guest though, as
> data should be cached already by ARC on the host, you're just going through
> an extra step of reading through the virtual disk driver, and into host
> ARC, instead of directly from the guest memory. Would need testing to know
> what performance was like and if there are any side effects.
>
> I do agree that it doesn't seem unnecessary to have any redundancy in the
> guest if the host pool is redundant. Save for any glaring bugs in the
> virtual disk emulation, you wouldn't expect to get errors on the guest pool
> if the host pool is already checksumming the data.
>
> It's also worth testing with guest ARC enabled but just limited to a
> fairly small size, so you're not disabling it entirely, but doing at little
> double-caching as possible.
>
> ZFS features seems perfect for virtual hosts, although it's not ideal that
> you have to give up a big chunk of host RAM for ARC. You may also find that
> you need to limit host ARC, then only use "MAX_RAM - MY_ARC_LIMIT" for
> guests. Otherwise you'll have ZFS and VMs fighting for memory and enough of
> us have seen what shouldn't, but unfortunately does happen in that
> situation.
>
> Matt
> -
>
> > Marcus
>

I've tried this in the past, and found the worst performance penalty was
with ARC disabled in guest. I tried with ARC enabled on host and guest,
only on host, only on guest. There was a significant performance penalty
with either ARC disabled.

I'd still recommend to experiment with it on your own to see if the hit is
acceptable or not.

Shameless plug: I'm working on a project (tunnelfs.io) which should be
useful for this use case. :) Unfortunately, there is no ETA on usable code
yet.

--
Marie Helene Kvello-Aune