Possible way to distribute NFS?

mike mike503 at gmail.com
Mon Nov 28 04:05:05 GMT 2005

(Note: also sent this to -fs, but it's relatively low traffic, and a
lot of people on -hackers have the knowledge of whether or not this is
even an idea worth giving a shot)

I've been looking for clustered FS or (if I have to) clustered NFS
solutions, and there's not many out there. Unless you want to go with
an expensive appliance. Currently you can't "roll your own" at all
with FreeBSD.

However, I think I've figured out a way that might work; basically
using the memcached[1] idea of hashing the requests to identify which
server is serving the data could alleviate any issues with NFS not
being "globally" aware, and allow for multiple NFS servers to read
from the same physical shared storage (note that the expectation is
the shared storage is being accessed from multiple physical servers,
and redundancy is done below the actual FS layer)

If someone could write an NFS client using FUSE (perhaps?) and have it
integrate with memcached[1], couldn't that basically allow for n+1 scaling
out of NFS servers and storage? Basically any time the FS would need
to issue or check a file lock, it would hit the memcache API to check
if it's been locked or "represented" yet by a specific NFS server.

(Note: whether or not it actually needs a "lock" or it's just saying
"I am NFS server #1 and /foo/bar/baz.txt will be served from my
server" is something I'd let someone smarter figure out - perhaps just
taking ownership of files regardless of the need to lock it would

I'm just brain dumping here. Note that I am not that advanced of a
programmer (I'm a PHP scripter) so filesystem and hardware I/O
semantics are not my specialty; this is just what I've got from
Googling until I'm blue in the face and why multi-path storage
solutions are still not very cheap/open source (only a couple
possibilities, each with limitations and caveats...)

Perhaps someone could shed some light, tell me it's stupid, or tell me
it's worthwhile?

NFS is pretty much tried and true, adding a small middle layer in
between to allow for distribution of NFS requests and alleviating the
lock conflicts might be a much more usable solution? Perhaps this
could fill the void of no GFS for FreeBSD?

My idea for design would be N number of NFS servers behind a single
virtual IP (or this "middleware" layer would be listening on this
virtual IP intercepting all the NFS requests and routing them
appropriately to the NFS servers behind them - also, for all intents
and purposes, this middleware daemon *could* just be on the same
physical machine as each NFS server - saving the need for separate
middleware servers too)


[1] "memcached" is a distributed memory caching system -
http://www.danga.com/memcached/ - it can be scaled up or out, and can
handle failover inside of the protocol (if a memcached server
disappears, it's info is forgotten and freed up for another active
memcached server)

More information about the freebsd-hackers mailing list