High traffic NFS performance and availability problems

[Paul J. Pathiakis]

On Thursday 17 February 2005 19:36, David Rice wrote:
> We are a web hosting company that runs  exclusivly on FreeBSD. We are
> having storage availability and performance problems. All of our storage is
> exported via NFS to the client machines Any suggestions or advice will be
> greatly appreciated. We are willing to pay someone on a consulting basis to
> help us solve these problems. Please email me off list if you are a
> consultant.
>

Hi David,

	First off, I'm trying to understand your configuration.  Do you have all your 
client's content on your server(s) and you export it to the client web server 
boxen?  This is my assumption.
	Check on your nfsstat's to see what's going on with the clients and servers.
	I'm a big fan of proper planning out storage on systems and keeping things 
flying at the local disk speed.  My use of NFS is for administration and user 
accounts in networked environment, not in something like a web farm.
	You may want to consider a paradigm shift.

Options:
1)	I'm not sure of the current state of this, but I know all the storage 
vendors are moving in this direction...  iSCSI or rSCSI.  I've seen some 
efforts and momentum on this in the past.  It's basically SAN on cheap 
hardware.  It runs via encapsulating SCSI in IP and runs over a simple 
network of switches and cat5 along with iSCSI cards that can be inserted into 
large disk repository servers.

2)  I used to work at one of the largest web farms, Genuity/BBNPlanet.  The 
simplest thing was to work was on Jumpstart to create cookie cutter boxen.  
Mirror hard drives for reliability, connect lots of disks, with redundancy, 
cluster.  (This can all be done with FreeBSD right now.  Check into PXE 
booting, gmirror <-this rocks, build large machines with redundant 
controllers, cluster using pf/altq/CARP)  Use iSCSI if you need to, but, more 
than likely, if you plan intelligently, you can make great use of local 
storage and virtual hosts with Apache.  As you grow, plug in more disks for 
customers or sharing.  Backup the whole thing with a centralized backup 
system using Bacula (configure Bacula as part of jumpstart so you don't have 
to screw with it too much)

3)  Check the performance numbers and tunings of NFS on various OS, and tune 
the heck out of it.  There's tons of numbers out there, but I don't have them 
offhand.  Also, with the advent of the new ATA code (if you're using 
ATA's/SATA for storage, SCSI would be better), the new vfs code that is now 
Giant Free, 5.4 is going to be some serious kickass OS on the underlying IO 
before getting to the NFS.  (If your data is read-only, I believe that helps 
a lot by exporting it read-only.  Of course, I'm not sure how out of date my 
info is, I'm building a startup right now.)

Just trying to help,

Paul Pathiakis