Cluster Filesystem for FreeBSD - any interest?

[Eric Anderson]

Allan Fields wrote:
> On Tue, Jun 21, 2005 at 09:35:56AM -0500, Eric Anderson wrote:
> 
>>This is something I've brought up before on other lists, but I'm curious 
>>if anyone is interested in developing a BSD licensed clustered 
>>filesystem for FreeBSD (and anyone else)?
> 
> 
> A few questions:
> 
> Could this be done as a stackable file system (vnode layer distributed
> file system) or did you have something else in mind (i.e. specifically
> a full implementation of a network filesystem including storage
> layer)?

Hmm.  I'm not sure if it can or not.  I'll try to explain what I'm 
dreaming of.  I currently have about 1000 clients needing access to the 
same pools of data (read/write) all the time.  The data changes 
constantly.  There is a lot of this data.  We use NFS currently. 
FreeBSD is *very* fast and stable at serving NFS data.  The problem is, 
that even though it is very fast and stable, I still cannot pump out 
enough bits fast enough with one machine, and if that one machine fails 
(hardware problems, etc), then all my machines are hung waiting for me 
to bring it back online.

So, what I would love to have, is this kind of setup: shared media 
storage (fibre channel SAN, iscsi, or something like ggated possibly), 
connected up to a cluster of hosts running FreeBSD.  Each FreeBSD server 
has access to the logical disks, same partitions, and can mount them all 
r/w.  Now, I can kind of do this now, however there are obviously some 
issues with this currently.  I want all machines in this cluster to be 
able to serve the data via NFS (or http, or anything else for that 
matter really - if you can make NFS work, anything will pretty much 
work) simultaneously from the same partitions, and see writes 
immediately as the other hosts in the cluster commit them.

I currently have a solution just like this for Linux - Polyserve 
(http://www.polyserve.com) has a clustered filesystem for linux, that 
works very well.  I've even tried to convince them to port it to 
FreeBSD, but it falls on deaf ears, so it's time to make our own.


> Why not a port of an existing network filesystem say from Linux?
> (A BSD rewrite could be done, if the code was GPLed.)  Would
> cross-platform capabilities make sense?

That would work fine I'm sure - but I have found some similar threads in 
the past that claim it would be just as hard and time consuming to port 
one as it would be to create one from scratch.   Cross platform 
capabilities would be great, but I'm mostly interested in getting 
FreeBSD into this arena (as it will soon be an extremely important one 
to be in).


> How do you see this comparing to device-level solutions?  I know
> the argument can be made to implement file systems/storage
> abstractions at multiple layers, but I thought I might ask.

I'm not sure of a device level solution that does this.  I think the OS 
has to know to commit the meta-data to a journal, or otherwise let the 
other machines know about locking, etc, in order for this to work.


> The other thing is there a wealth of filesystem papers out there,
> any in specific caught your eye?

No - can you point me to some?

I'll be honest here - I'm not a code developer.  I would love to learn 
some C here, and 'just do it', but filesystems aren't exactly simple, so 
I'm looking for a group of people that would love to code up something 
amazing like this - I'll support the developers and hopefully learn 
something in the process.  My goal personally would be to do anything I 
could to make the developers work most productively, and do testing.  I 
can probably provide equipment, and a good testbed for it.

Eric



-- 
------------------------------------------------------------------------
Eric Anderson        Sr. Systems Administrator        Centaur Technology
A lost ounce of gold may be found, a lost moment of time never.
------------------------------------------------------------------------