Possible way to distribute NFS?

[Chuck Lever]

mike-

you want pNFS.  this is something that is currently being worked out by 
the IETF, various NFS vendors, and members of the open source community. 
  google for pNFS or Parallel NFS.

mike wrote:
> I've been looking for clustered FS or (if I have to) clustered NFS
> solutions, and there's not many out there. Unless you want to go with
> an expensive appliance. Currently you can't "roll your own" at all
> with FreeBSD.
> 
> However, I think I've figured out a way that might work; basically
> using the memcached[1] idea of hashing the requests to identify which
> server is serving the data could alleviate any issues with NFS not
> being "globally" aware, and allow for multiple NFS servers to read
> from the same physical shared storage (note that the expectation is
> the shared storage is being accessed from multiple physical servers,
> and redundancy is done below the actual FS layer)
> 
> If someone could write an NFS client using FUSE (perhaps?) and have it
> integrate with memcached[1], couldn't that basically allow for n+1 scaling
> out of NFS servers and storage? Basically any time the FS would need
> to issue or check a file lock, it would hit the memcache API to check
> if it's been locked or "represented" yet by a specific NFS server.
> 
> (Note: whether or not it actually needs a "lock" or it's just saying
> "I am NFS server #1 and /foo/bar/baz.txt will be served from my
> server" is something I'd let someone smarter figure out - perhaps just
> taking ownership of files regardless of the need to lock it would
> work?)
> 
> I'm just brain dumping here. Note that I am not that advanced of a
> programmer (I'm a PHP scripter) so filesystem and hardware I/O
> semantics are not my specialty; this is just what I've got from
> Googling until I'm blue in the face and why multi-path storage
> solutions are still not very cheap/open source (only a couple
> possibilities, each with limitations and caveats...)
> 
> Perhaps someone could shed some light, tell me it's stupid, or tell me
> it's worthwhile?
> 
> NFS is pretty much tried and true, adding a small middle layer in
> between to allow for distribution of NFS requests and alleviating the
> lock conflicts might be a much more usable solution? Perhaps this
> could fill the void of no GFS for FreeBSD?
> 
> My idea for design would be N number of NFS servers behind a single
> virtual IP (or this "middleware" layer would be listening on this
> virtual IP intercepting all the NFS requests and routing them
> appropriately to the NFS servers behind them (also, for all intents
> and purposes, this middleware daemon *could* just be on the same
> physical machine as each NFS server - saving the need for separate
> middleware servers too)
> 
> Thanks,
> mike
> 
> [1] "memcached" is a distributed memory caching system -
> http://www.danga.com/memcached/ - it can be scaled up or out, and can
> handle failover inside of the protocol (if a memcached server
> disappears, it's info is forgotten and freed up for another active
> memcached server)