MooseFS on FreeBSD (was: HAST + ZFS + NFS + CARP)

[Joe Love]

On Aug 17, 2016, at 8:33 AM, krad <kraduk at gmail.com> wrote:
> 
> What are peoples experiences on running something like moosfs on top of
> zfs? It looks really compelling on certain levels, but i'm not sure about
> the reality in a production network yet.
> 

I did some experimenting with MooseFS on a test cluster (using ZFS as the local storage on the nodes).  That was nearly a year ago, when I decided it wasn’t a good fit as a storage backend to vmware (primarily because of the overhead involved with traversing from vmware over nfs to moosefs).

They did a bunch of tweaking back then as I kept prodding a bit for better operations on FreeBSD, but they ultimately ran into a stumbling block with I/O caching in FreeBSD.

Here is their analysis of what they ran into, and the solution they came up with for higher throughput:
https://sourceforge.net/p/moosefs/mailman/message/34483159/ <https://sourceforge.net/p/moosefs/mailman/message/34483159/>

I’d love to get around to testing MooseFS as a backing store for bhyve, but I currently lack the power & network ports to connect up my test nodes again.

-Joe