Cluster Filesystem for FreeBSD - any interest?

[Eric Anderson]

yf-263 wrote:
> 在 2005-06-22三的 07:09 -0500，Eric Anderson写道：
> 
>>Allan Fields wrote:
>>
>>>On Tue, Jun 21, 2005 at 09:35:56AM -0500, Eric Anderson wrote:
>>>
>>>
>>>>This is something I've brought up before on other lists, but I'm curious 
>>>>if anyone is interested in developing a BSD licensed clustered 
>>>>filesystem for FreeBSD (and anyone else)?
>>>
>>>
>>>A few questions:
>>>
>>>Could this be done as a stackable file system (vnode layer distributed
>>>file system) or did you have something else in mind (i.e. specifically
>>>a full implementation of a network filesystem including storage
>>>layer)?
>>
>>Hmm.  I'm not sure if it can or not.  I'll try to explain what I'm 
>>dreaming of.  I currently have about 1000 clients needing access to the 
> 
> 
> This is exactly what we are doing now :) but only a proof-of-concept
> toy.
> 
> Now it has a asymmetric arch, i.e. a client, some metadata server, and
> some filedata server.
> 
> thanks for your professional description as what we are target to ;)
> 
> 
>>same pools of data (read/write) all the time.  The data changes 
>>constantly.  There is a lot of this data.  We use NFS currently. 
>>FreeBSD is *very* fast and stable at serving NFS data.  The problem is, 
>>that even though it is very fast and stable, I still cannot pump out 
>>enough bits fast enough with one machine, and if that one machine fails 
>>(hardware problems, etc), then all my machines are hung waiting for me 
>>to bring it back online.
>>
>>So, what I would love to have, is this kind of setup: shared media 
>>storage (fibre channel SAN, iscsi, or something like ggated possibly), 
>>connected up to a cluster of hosts running FreeBSD.  Each FreeBSD server 
>>has access to the logical disks, same partitions, and can mount them all 
>>r/w.  Now, I can kind of do this now, however there are obviously some 
>>issues with this currently.  I want all machines in this cluster to be 
>>able to serve the data via NFS (or http, or anything else for that 
>>matter really - if you can make NFS work, anything will pretty much 
>>work) simultaneously from the same partitions, and see writes 
>>immediately as the other hosts in the cluster commit them.
>>
>>I currently have a solution just like this for Linux - Polyserve 
>>(http://www.polyserve.com) has a clustered filesystem for linux, that 
>>works very well.  I've even tried to convince them to port it to 
>>FreeBSD, but it falls on deaf ears, so it's time to make our own.
>>
>>
>>
>>>Why not a port of an existing network filesystem say from Linux?
>>>(A BSD rewrite could be done, if the code was GPLed.)  Would
>>>cross-platform capabilities make sense?
>>
>>That would work fine I'm sure - but I have found some similar threads in 
>>the past that claim it would be just as hard and time consuming to port 
>>one as it would be to create one from scratch.   Cross platform 
>>capabilities would be great, but I'm mostly interested in getting 
>>FreeBSD into this arena (as it will soon be an extremely important one 
>>to be in).
>>
>>
>>
>>>How do you see this comparing to device-level solutions?  I know
>>>the argument can be made to implement file systems/storage
>>>abstractions at multiple layers, but I thought I might ask.
>>
>>I'm not sure of a device level solution that does this.  I think the OS 
>>has to know to commit the meta-data to a journal, or otherwise let the 
>>other machines know about locking, etc, in order for this to work.
>>
>>
>>
>>>The other thing is there a wealth of filesystem papers out there,
>>>any in specific caught your eye?
>>
>>No - can you point me to some?
>>
>>I'll be honest here - I'm not a code developer.  I would love to learn 
>>some C here, and 'just do it', but filesystems aren't exactly simple, so 
>>I'm looking for a group of people that would love to code up something 
>>amazing like this - I'll support the developers and hopefully learn 
>>something in the process.  My goal personally would be to do anything I 
>>could to make the developers work most productively, and do testing.  I 
>>can probably provide equipment, and a good testbed for it.

You've mentioned this a few times before - is this something you will be 
offering publicly sometime soon?

Eric




-- 
------------------------------------------------------------------------
Eric Anderson        Sr. Systems Administrator        Centaur Technology
A lost ounce of gold may be found, a lost moment of time never.
------------------------------------------------------------------------