glusterfs + FreeBSD [was Re: CEPH + FreeBSD]

[justin at postgresql.org]

[ adding Jeff and Kaleb (Gluster folk) ]

(remainder of the reply is at the bottom)

On 2015-09-07 06:13, Jordan Hubbard wrote:
> [ Adding Justin Clift to CC line ]
>
>> On Sep 6, 2015, at 3:18 PM, Rick Macklem <rmacklem at uoguelph.ca> 
>> wrote:
>>
>> Hmm. From a quick look at their web page (I looked once before as 
>> well), I don't
>> think RiakCS has what I need to do pNFS in a reasonable (for me) 
>> amount of effort.
>> Two things that glusterFS has that I am hoping to use (and I don't 
>> think RiakCS has
>> either of these) are:
>> - A Fuse file system interface which allows the kernel nfsd to 
>> access the store as
>>  a file system, so that it can provide the metadata services (NFS 
>> without the reads/writes).
>> - A userland NFSv3 server in each node which will allow the node to 
>> act as a data server.
>>
>> If I am wrong and RiakCS does support a VFS file system interface 
>> (via Fuse or ???), then
>> please correct me.
>
> You are not wrong.  RiakCS is basically just a distributed object
> store with an Amazon S3 compatible API.  It doesn’t attempt to expose
> the object store as a filesystem or provide any other modes of access
> to it other than via S3 or its own “database API” since it’s also (at
> its heart) a distributed database.
>
> In all fairness to Basho, I also think that was probably the right
> call.  It’s not a file system and doesn’t pretend to be one, it’s 
> just
> a distributed database that can store arbitrary numbers (and sizes) 
> of
> objects in a cluster with fault-tolerance, multi-data center tenancy
> and so on (I also have no connection with Basho so if it sounds like
> I’m trying to sales pitch it or something, I’m not, I’m simply trying
> to be accurate in describing it given that we’re also evaluating it 
> as
> a possible technology to incorporate into FreeNAS, for which the 
> topic
> of S3-compatible object storage comes up fairly frequently.
>
> Now glusterfs is another kettle of fish, and like I said, I think the
> bigger question there is just what FreeBSD’s relationship with it is.
> The https://wiki.freebsd.org/GlusterFS wiki was created over a year
> ago and seems to be suffering from an overall lack of specific goals.
> Does the FreeBSD project want glusterfs and/or is there anyone in the
> FreeBSD community even interested in deploying it in a production
> scenario?  Is anyone willing to actually maintain a port, just as the
> minimum price of admission?
>
> The bug tracking this, namely
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=194409, has been
> open for almost a year now and while it looks like ‘craig001” has 
> been
> trying fairly hard to get 3.7.2 up and working, he’s hitting some
> walls and it’s not even clear he’s in contact with the right folks.
>
> As the previous glusterfs mail thread indicated, I was also happy to
> add an earlier version of his port to the FreeNAS repo and play with
> it there, though I was forced to take it back out of FreeNAS when we
> ran into a number of issues just trying to make it work with the 
> Linux
> glusterfs reference VMs we set up.  Should the underlying port get
> some love again, it would be the simplest thing in the world for me 
> to
> refresh our copy of it and uncomment the line which adds glusterfs to
> the FreeNAS manifest (and being a storage appliance, it’s rather of 
> an
> obvious test bed for this since people running FreeNAS are also the
> target audience for technologies like this).
>
> If getting glusterfs up as a stable, working substrate is also a
> prerequisite for getting pNFS, then I think a lot of us in the 
> storage
> industry would be the first to say “What do we have to do?  How can 
> we
> help?”   I already volunteered to send Rick an AMD64 architecture
> machine for development, but I suspect that’s actually the least of
> our obstacles right now. :-)

A couple of points... ;)

* We submitted a patch for FreeBSD's fuse... that seems to have just 
sat,
   and sat, and sat... waiting for review and gone nowhere.  That would 
be
   good to get fixed. (just saying)

* Our guy who was putting in most of the effort for cross platform 
porting
   (Harsha), left Red Hat.  Just added him to CC anyway, in case he has
   insight. :)

* I also left Red Hat.  I'm still doing a bit of stuff with Gluster, 
but
   that's likely to taper off after I finish getting the new "Gluster 
Forge"
   project done.

Hopefully Niels (he knows Gluster NFS) and Kaleb (Gluster greybeard ;>)
can move this forward.

Regards and best wishes,

Justin Clift