pNFS server Plan B

[Jordan Hubbard]

> On Jun 18, 2016, at 6:14 PM, Chris Watson <bsdunix44 at gmail.com> wrote:
> 
> Since Jordan brought up clustering, I would be interested to hear Justin Gibbs thoughts here. I know about a year ago he was asked on an "after hours" video chat hosted by Matt Aherns about a feature he would really like to see and he mentioned he would really like, in a universe filled with time and money I'm sure, to work on a native clustering solution for FreeBSD. I don't know if he is subscribed to the list, and I'm certainly not throwing him under the bus by bringing his name up, but I know he has at least been thinking about this for some time and probably has some value to add here. 

I think we should also be careful to define our terms in such a discussion.  Specifically:

1. Are we talking about block-level clustering underneath ZFS (e.g. HAST or ${somethingElse}) or otherwise incorporated into ZFS itself at some low level?  If you Google for “High-availability ZFS” you will encounter things like RSF-1 or the somewhat more mysterious Zetavault (http://www.zeta.systems/zetavault/high-availability/) but it’s not entirely clear how these technologies work, they simply claim to “scale-out ZFS” or “cluster ZFS” (which can be done within ZFS or one level above and still probably pass the Marketing Test for what people are willing to put on a web page).

2. Are we talking about clustering at a slightly higher level, in a filesystem-agnostic fashion which still preserves filesystem semantics?

3. Are we talking about clustering for data objects, in a fashion which does not necessarily provide filesystem semantics (a sharding database which can store arbitrary BLOBs would qualify)?

For all of the above:  Are we seeking to be compatible with any other mechanisms, or are we talking about a FreeBSD-only solution?

This is why I brought up glusterfs / ceph / RiakCS in my previous comments - when talking to the $users that Rick wants to involve in the discussion, they rarely come to the table asking for “some or any sort of clustering, don’t care which or how it works” - they ask if I can offer an S3 compatible object store with horizontal scaling, or if they can use NFS in some clustered fashion where there’s a single namespace offering petabytes of storage with configurable redundancy such that no portion of that namespace is ever unavailable.

I’d be interested in what Justin had in mind when he asked Matt about this.  Being able to “attach ZFS pools to one another” in such a fashion that all clients just see One Big Pool and ZFS’s own redundancy / snapshotting characteristics magically apply to the überpool would be Pretty Cool, obviously, and would allow one to do round-robin DNS for NFS such that any node could serve the same contents, but that also sounds pretty ambitious, depending on how it’s implemented.

- Jordan