High traffic NFS performance and availability problems

[Eric Anderson]

David Rice wrote:
> We are a web hosting company that runs  exclusivly on FreeBSD. We are having 
> storage availability and performance problems. All of our storage is exported
> via NFS to the client machines Any suggestions or advice will be greatly 
> appreciated. We are willing to pay someone on a consulting basis to help us 
> solve these problems. Please email me off list if you are a consultant. 

I have lots of heavily abused NFS servers running FreeBSD (I usually have 400-500 3.XGHz P4's hammering each NFS server),  so maybe I can help a bit.

> On the file server side we have:
> 
> Dell PowerEdge 1750's
> Dell Perc4 RAID controller
> Dell Power Vault 220 storage shelf.
> (12) 146GB SCSI drives on one SCSI Bus with a hot spare (~1.3TB file system)
> GigaBit ethernet to the NFS client machines
> 1GB RAM
> (2) 2.4 GHZ Xenon Processors
> FreeBSD 5.2.1 or FreBSD 5.3

If you are on the 5.x branch, make sure you are running -STABLE (cvsup and buildworld if you can).  Minimum 5.3-RELEASE.


> On the client side we have: 
> 
> Dell PowerEdge 1750's
> 1GB RAM
> (2) 2.4 GHZ Xenon Processors
> GigaBit ethernet
> 36GB SCSI root disk
> FreeBSD 4.9 to FreeBSD 4.11
> 
> On the network side we have:
> Gigabit ethernet
> Foundry BigIron and NetIron swithes
> Cisco 6509 with Gigabit switch blades 
> 
> 
> Typicly we have 7 client boxes mounting storage from a single file server. 
> Each client box servers 1000 web sites and associate email. We have
> done the basic NFS tuning (ie: Read write size optimization and kernel tuning)
> 
> 
> The problems we are having is as follows.
> 
> 1. Slow perfomance during peek traffic periods
> 2. Client boxes have high load averages and sometimes crashes due to slow NFS 
> performance.
> 3. File servers that randomly crash with "Fatal trap 12: page fault while in 
> kernel mode"
> 4. With soft updates enabled during FSCK the fileserver will freeze with all 
> NFS processs in the "snaplck" state. We disabled soft updates because of 
> this.
> 
> I can provide and other details about our configuration if needed. 

During peak performance, from the clients, how do you 'feel' the slowness?  
Does an 'ls' in the directory over NFS stall?  
How about an ls -al?  
What kind of authentication mechanism are you using?
How many nfsd processes do you have running on the server?
During peak performance, how many of those nfsd processes are in a state other than 'nfsd' (use top, ps, etc)?

Eric


-- 
------------------------------------------------------------------------
Eric Anderson        Sr. Systems Administrator        Centaur Technology
I have seen the future and it is just like the present, only longer.
------------------------------------------------------------------------