zfs on geli vs. geli on zfs (via zvol)

[Pete French]

> While zfs on geli is less complex (in the sense that geli on zfs
> involves two layers of filesystems), I'm concerned as to whether
> encrypting the device will somehow affect zfs' ability to detect
> silent corruption, self-heal, or in any other way adversely affect
> zfs' functionality.  In my mind, if I use geli on zfs, then I've
> got zfs directly on a device and the zvol it's providing will be
> transparently gaining the benefits of zfs' various features, providing
> a "safety layer" against device failure and silent corruption that
> I'm not sure if geli would detect.

These are very good questions - I ran ZFS on top of geli for a long time,
and what I found was that when there were problems with the underlying
discs, then geli would have problems and those would not be reported
back to ZFS properly. I got lockups under those circumstances - when
is witched to ZFS on top directly what I got were discs dropping out and
ZFS properly continuing with the remaining drives.

I never managed to characterise it well enougnh to file a PR I am
afraid though - it only ever happened with failing hardware which
made it hard to reproduce.

-pete.