9.0-RELEASE-p4 + NFS + ZFS = issues... :/ (probably a memory leak)

[Rick Macklem]

Xavier Beaudouin wrote:
> Hello there,
> 
> I try to make a FreeBSD ZFS NFS server with the following
> configuration
> :
> 
> - Supermicro X7DB8
> - 2 L5420
> - 12G RAM
> - 8 x 300G SCSI drives using the 2 U320 scsi channel.
> - lagg (LACP) of 2 e1000
> - zpool :
> 
> # zpool status
> pool: vol
> state: ONLINE
> scan: scrub repaired 0 in 2h11m with 0 errors on Thu Nov 1 07:42:05
> 2012
> config:
> 
> NAME STATE READ WRITE CKSUM
> vol ONLINE 0 0 0
> raidz1-0 ONLINE 0 0 0
> da0 ONLINE 0 0 0
> da4 ONLINE 0 0 0
> raidz1-1 ONLINE 0 0 0
> da1 ONLINE 0 0 0
> da5 ONLINE 0 0 0
> raidz1-2 ONLINE 0 0 0
> da2 ONLINE 0 0 0
> da6 ONLINE 0 0 0
> raidz1-3 ONLINE 0 0 0
> da3 ONLINE 0 0 0
> da7 ONLINE 0 0 0
> 
> 
> I have tried this patch : http://people.freebsd.org/~namei-leak.patch
> 
> And issue still here.
> 
> Also tried to move from newnfs to oldnfs server and client, same
> issue.
> 
> Issue is comming when copying from a netapp to this machine a 70G data
> with bunch of big files (eg 600M ~ 8G) and web files.
> 
> All NFS clients use NFSv3 and UDP.
> 
There is this patch for NFSv3 over UDP:
  http://people.freebsd.org/~rmacklem/udp-timer.patch
  (It applies to both NFS clients, since it is krpc related.)
I can't remember what the implications of not having this patch is, but
if the client retries NFS RPCs too agressively, it could use a lot of
DRC storage (mbuf clusters mainly) in the NFS server (if the NFS server
is a FreeBSD box).
OR
Try the mounts using TCP and see if the problem goes away.

If it is slow leak, doing "vmstat -z" and "vmstat -m" while the system
is getting constipated might show what is leaking.

> Tried :
> 
> nfs client netapp and freebsd on the same machine : 1srt rsync from
> netapp to freebsd ok : no issue, second rsync : memory leak, swap is
> used (about less than 100Mb, but this trigger the bug.
> 
> nfs client of the netapp is the "filer" machine, and the destination
> is
> local zfs spool. The rsync to make up to date data trigger as well the
> issue.
> 
I'll admit you lost me w.r.t. what is the NFS client and what is the
NFS server. (Does "netapp" refer to a Netapp Filer or a FreeBSD box?)
I'm not sure that matters, except that it isn't obvious to me if the
leak you are referring to is in the NFS client or NFS server?

> /boot/loader.conf
> zfs_load="YES"
> if_lagg_load="YES"
> autoboot_delay="10" # Delay in seconds before autobooting,
> start
> # http://icesquare.com/wordpress/how-to-improve-zfs-performance/
> #I have 8G of Ram
> #vfs.zfs.prefetch_disable=0
> 
> #If Ram = 4GB, set the value to 512M
> #If Ram = 8GB, set the value to 1024M
> #vfs.zfs.arc_min="1024M"
> 
> #Ram x 0.5 - 512 MB
> #vfs.zfs.arc_max="3584M"
> 
> #Ram x 2
> vm.kmem_size_max="24G"
> 
> #Ram x 1.5
> #vm.kmem_size="12G"
> 
> 
> /etc/sysctl.conf
> # $FreeBSD: release/9.0.0/etc/sysctl.conf 112200 2003-03-13 18:43:50Z
> mux $
> #
> # This file is read when going to multi-user and its contents piped
> thru
> # ``sysctl'' to adjust kernel values. ``man 5 sysctl.conf'' for
> details.
> #
> 
> # Uncomment this to prevent users from seeing information about
> processes that
> # are being run under another UID.
> #security.bsd.see_other_uids=0
> kern.maxfiles=64000
> vfs.nfsd.maxthreads=16
> vfs.zfs.prefetch_disable=1
> 
> 
> The latter (vfs.zfs.prefetch_disable=1) make the bug a bit strange.
> Inactive memory keeps growing, but active and free is still very low
> during the copy. Then machine use all of his memory, and NFS server
> not
> responsive.
> 
> /etc/rc.conf
> hostname="filernew"
> keymap="fr.iso.acc.kbd"
> cloned_interfaces="lagg0"
> ifconfig_em0="up"
> ifconfig_em1="up"
> ifconfig_lagg0="inet 172.31.x.x netmask 255.255.0.0 laggproto lacp
> laggport em0 laggport em1"
If you can take lagg out of the mix, I'd try that.

> defaultrouter="172.31.x.x"
> sshd_enable="YES"
> ntpd_enable="YES"
> powerd_enable="YES"
> # Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
> dumpdev="NO"
> zfs_enable="YES"
> puppet_enable="YES"
> # NFS
> rpcbind_enable="YES"
> nfs_server_enable="YES"
> nfs_server_flags="-u -t -n 4"
> mountd_flags="-r"
> mountd_enable="YES"
> rpc_lockd_enable="YES"
> rpc_statd_enable="YES"
> 
> # Added by Puppet
> bsnmpd_enable="YES"
> 
> Kind regards...
> 
> Xavier
> 
Good luck with it, rick

> --
> Xavier Beaudouin
> _______________________________________________
> freebsd-fs at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"