High traffic NFS performance and availability problems

[David Rice]

We are a web hosting company that runs  exclusivly on FreeBSD. We are having 
storage availability and performance problems. All of our storage is exported
via NFS to the client machines Any suggestions or advice will be greatly 
appreciated. We are willing to pay someone on a consulting basis to help us 
solve these problems. Please email me off list if you are a consultant. 

On the file server side we have:

Dell PowerEdge 1750's
Dell Perc4 RAID controller
Dell Power Vault 220 storage shelf.
(12) 146GB SCSI drives on one SCSI Bus with a hot spare (~1.3TB file system)
GigaBit ethernet to the NFS client machines
1GB RAM
(2) 2.4 GHZ Xenon Processors
FreeBSD 5.2.1 or FreBSD 5.3

On the client side we have: 

Dell PowerEdge 1750's
1GB RAM
(2) 2.4 GHZ Xenon Processors
GigaBit ethernet
36GB SCSI root disk
FreeBSD 4.9 to FreeBSD 4.11

On the network side we have:
Gigabit ethernet
Foundry BigIron and NetIron swithes
Cisco 6509 with Gigabit switch blades 


Typicly we have 7 client boxes mounting storage from a single file server. 
Each client box servers 1000 web sites and associate email. We have
done the basic NFS tuning (ie: Read write size optimization and kernel tuning)


The problems we are having is as follows.

1. Slow perfomance during peek traffic periods
2. Client boxes have high load averages and sometimes crashes due to slow NFS 
performance.
3. File servers that randomly crash with "Fatal trap 12: page fault while in 
kernel mode"
4. With soft updates enabled during FSCK the fileserver will freeze with all 
NFS processs in the "snaplck" state. We disabled soft updates because of 
this.

I can provide and other details about our configuration if needed. 

David Rice
drice at globat.com

Thank You.