Unstable NFS on recent CURRENT

[Paul Mather]

On Mar 7, 2016, at 9:55 PM, Rick Macklem <rmacklem at uoguelph.ca> wrote:

> Paul Mather (forwarded by Ronald Klop) wrote:
>> On Sun, 06 Mar 2016 02:57:03 +0100, Paul Mather <paul at gromit.dlib.vt.edu>
>> wrote:
>> 
>>> On my BeagleBone Black running 11-CURRENT (r296162) lately I have been
>>> having trouble with NFS.  I have been doing a buildworld and buildkernel
>>> with /usr/src and /usr/obj mounted via NFS.  Recently, this process has
>>> resulted in the buildworld failing at some point, with a variety of
>>> errors (Segmentation fault; Permission denied; etc.).  Even a "ls -alR"
>>> of /usr/src doesn't manage to complete.  It errors out thus:
>>> 
>>> =====
>>> [[...]]
>>> total 0
>>> ls: ./.svn/pristine/fe: Permission denied
>>> 
>>> ./.svn/pristine/ff:
>>> total 0
>>> ls: ./.svn/pristine/ff: Permission denied
>>> ls: fts_read: Permission denied
>>> =====
>>> 
>>> On the console, I get the following:
>>> 
>>> newnfs: server 'chumby.chumby.lan' error: fileid changed. fsid
>>> 94790777:a4385de: expected fileid 0x4, got 0x2. (BROKEN NFS SERVER OR
>>> MIDDLEWARE)
>>> 
>>> 
>>> I am using a FreeBSD/amd64 10.3-PRERELEASE (r296412) as the NFS server.
>>> On the BeagleBone Black, I am mounting /usr/src and /usr/obj via
>>> /etc/fstab as follows:
>>> 
>>> chumby.chumby.lan:/build/src/head /usr/src nfs rw,nfsv4 0 0
>>> chumby.chumby.lan:/build/obj/bbb /usr/obj nfs rw,nfsv4 0 0
>>> 
>>> 
>>> /build/src/head and /build/obj/bbb are both ZFS file systems.
>>> 
> Is it possible that a ZFS file system has gotten to the point where the
> i-node# exceeds 32bits? ZFS does support more than 32bits for i-node#s,
> but FreeBSD does not (it truncates to the low order 32bits).
> I know diddly about ZFS, so I don't know if you actually have to create
> more than 4billion files to get the i-node# to exceed 32bits or ???
> 
> There has been work done on making ino_t 64bits, but it hasn't made it
> into FreeBSD-current and I have no idea when it might.
> 
> If you could try a build on newly created file systems (or UFS ones
> instead of ZFS), that would tell you if the above might be the problem.


I don't think I have that big of a ZFS pool (it's 2 TB). :-)

It doesn't seem that there are excessive numbers of inodes, and the counts match up between the NFS client and server sides.

In the information below, chumby is the NFS server and beaglebone the client:

pmather at beaglebone:~ % mount
/dev/mmcsd0s2a on / (ufs, local, noatime, soft-updates)
devfs on /dev (devfs, local)
/dev/mmcsd0s1 on /boot/msdos (msdosfs, local, noatime)
tmpfs on /tmp (tmpfs, local)
tmpfs on /var/log (tmpfs, local)
tmpfs on /var/tmp (tmpfs, local)
chumby.chumby.lan:/build/src/head on /usr/src (nfs, nfsv4acls)
chumby.chumby.lan:/build/obj/bbb on /usr/obj (nfs, nfsv4acls)
pmather at beaglebone:~ % df -i /usr/src /usr/obj
Filesystem                        1K-blocks    Used     Avail Capacity iused      ifree %iused  Mounted on
chumby.chumby.lan:/build/src/head   2097152 1344484    752668    64%  147835    1505336    9%   /usr/src
chumby.chumby.lan:/build/obj/bbb  530875884 1949364 528926520     0%   70814 1057853040    0%   /usr/obj


paul at chumby:/home/paul> df -i /build/src/head /build/obj/bbb
Filesystem                  1K-blocks    Used     Avail Capacity iused      ifree %iused  Mounted on
zroot/SHARED/build/src/head   2097152 1344484    752668    64%  147835    1505336    9%   /build/src/head
zroot/SHARED/build/obj/bbb  530876268 1949364 528926904     0%   70814 1057853808    0%   /build/obj/bbb


On the NFS client system, these are the only NFS-related settings I have in /etc/rc.conf:

nfsuserd_enable="YES"
nfscbd_enable="YES"


Would you recommend I try it with nfscbd_enable="NO"?

I will try NFS from other clients to see whether it's just this FreeBSD/arm system that's having problems.

Cheers,

Paul.


> 
> rick
> 
>>> Has anyone else encountered this?  It has only started happening
>>> recently for me, it seems.  Prior to this, I have been able to do a
>>> buildworld and buildkernel successfully over NFS.
>>> 
>>> Cheers,
>>> 
>>> Paul.
>> 
>> I cc this to freebsd-fs for you.
>> 
>> Ronald.
>> _______________________________________________
>> freebsd-fs at freebsd.org mailing list
>> https://lists.freebsd.org/mailman/listinfo/freebsd-fs
>> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"
>> 
> _______________________________________________
> freebsd-arm at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-arm
> To unsubscribe, send any mail to "freebsd-arm-unsubscribe at freebsd.org"