NFS Server keeps timing out on a client?

[Kaya Saman]

Hi,

this is a really difficult one to diagnose :-(

The situation is this:

I have a server which I'm trying to serve up a home directory for. So 
when a user logs in they will be taken to /mnt/home/<user> NFS mount.

The home directory is a zfs filesystem stored on a ZPOOL.

The server runs FreeBSD 10.1 64bit and is powered by a quad core Intel 
Xeon with 8GB RAM and is connected to the network using a 4GbE lagg trunk.


The client side is Arch Linux, 64 bit connected via gigabit link.


They are on different vlans, and the network uses Jumbo MTU's however, I 
have also tested with regular mtu size of 1500 and same thing seems to 
be happening.


As the firewall/router also runs *BSD I tried to see what was happening 
using: tcpdump -eni (vlan) host <ip>


During the timeout periods I keep seeing this message:

2042: truncated-ip - 4 bytes missing!192.168.20.30.981 > 192.168.1.80.2049: xid 0x3b14188c (NFSv3) 1976 write [|nfs] (DF)2042: truncated-ip - 4 bytes missing!192.168.20.30.981 > 192.168.1.80.2049: xid 0x3b14188c (NFSv3) 1976 write [|nfs] (DF)


2042: truncated-ip - 4 bytes missing!<client_ip>.981 > <server_ip>.2049: 
xid 0x3b14188c (NFSv3) 1976 write [|nfs] (DF)


Unfortunately setting the "debug" flag in mountd on FreeBSD doesn't show 
any thing other then "mount ok".


On the client side I have put NFS into debug mode and am seeing this 
through dmesg:

[24012.265800] NFS: nfs_update_inode(0:41/1653343 fh_crc=0x9257f59a ct=1 
info=0x27e7f)
[24012.266207] NFS: nfs_update_inode(0:41/1653343 fh_crc=0x9257f59a ct=1 
info=0x27e7f)
[24012.266215] NFS: permission(0:41/1653343), mask=0x26, res=0
[24012.284651] NFS: nfs_update_inode(0:41/1653339 fh_crc=0x998992f7 ct=1 
info=0x7feff)
[24012.285181] NFS: nfs_update_inode(0:41/1653339 fh_crc=0x998992f7 ct=1 
info=0x7feff)
[24012.285251] NFS: 
dentry_delete(entries/ADDBB8F380DD96E0D9B49F9AF600706F6A1431BA, 4808cc)
[24012.294851] NFS: nfs_update_inode(0:41/1653196 fh_crc=0x6513c0a6 ct=1 
info=0x7feff)
[24012.295399] NFS: nfs_update_inode(0:41/1653196 fh_crc=0x6513c0a6 ct=1 
info=0x7feff)
[24012.295510] NFS: 
dentry_delete(entries/9AEB965A3D7D6F96E61BD8427D8296215AE6CE13, 4808cc)
[24013.731223] NFS: permission(0:41/4), mask=0x1, res=0
[24013.731237] NFS: permission(0:41/1649925), mask=0x1, res=0
[24013.731244] NFS: permission(0:41/1649944), mask=0x1, res=0
[24013.731251] NFS: dentry_delete(.config/menus, 808cc)


Has anyone experienced this type of behavior before, and possibly help 
guide me on how to debug it to get to the root of the problem?


Many thanks!


Regards,


Kaya