releng/13 release/13.0.0 : odd/incorrect diff result over nfs (in a zfs file systems context)

Rick Macklem rmacklem at uoguelph.ca
Fri May 21 00:55:21 UTC 2021


Mark Millard wrote:
>[I warn that I'm a fairly minimal user of NFS
>mounts, not knowing all that much. I'm mostly
>reporting this in case it ends up as evidence
>via eventually matching up with others observing
>possibly related oddities.]
>
>I got the following odd sequence (that I've
>mixed notes into). It involved a diff -r over NFS
>showing differences (files missing) and then a
>later diff finding matches for the same files,
>no file system changes made on either machine.
>I'm unable to reproduce the oddity on demand.
>
>Note: A larger scope diff -r originally returned the
>below as well, but doing the narrower diff -r did
>repeat the result and that is what I show. (I
>make no use of devel/ice .)
>
># diff -r /usr/ports/devel/ice/files /mnt/devel/ice/files | more
>Only in /usr/ports/devel/ice/files: Make.rules.FreeBSD
>Only in /usr/ports/devel/ice/files: patch-config-Make.common.rules
>Only in /usr/ports/devel/ice/files: patch-cpp-Makefile
>Only in /usr/ports/devel/ice/files: patch-cpp-allTests.py
>Only in /usr/ports/devel/ice/files: patch-cpp-config-Make.rules
>Only in /usr/ports/devel/ice/files: patch-cpp-include-Ice-FactoryTableInit.h
>Only in /usr/ports/devel/ice/files: patch-cpp-include-IceUtil-Config.h
>Only in /usr/ports/devel/ice/files: patch-cpp-include-IceUtil-ScannerConfig.h
>Only in /usr/ports/devel/ice/files: patch-cpp-src-Glacier2CryptPermissionsVerifier-CryptPermissionsVerifierI.cpp
>Only in /usr/ports/devel/ice/files: patch-cpp-src-Ice-ProxyFactory.cpp
>Only in /usr/ports/devel/ice/files: patch-cpp-src-IceGrid-PluginFacadeI.cpp
>Only in /usr/ports/devel/ice/files: patch-cpp-src-IceGrid-RegistryI.cpp
>Only in /usr/ports/devel/ice/files: patch-cpp-src-IceSSL-Makefile
>Only in /usr/ports/devel/ice/files: patch-cpp-test-Glacier2-ssl-Server.cpp
>Only in /usr/ports/devel/ice/files: patch-cpp-test-Glacier2-staticFiltering-run.py
>Only in /usr/ports/devel/ice/files: patch-cpp-test-Ice-info-AllTests.cpp
>Only in /usr/ports/devel/ice/files: patch-cpp-test-Ice-metrics-AllTests.cpp
>Only in /usr/ports/devel/ice/files: patch-cpp-test-Ice-objects-Makefile
>Only in /usr/ports/devel/ice/files: patch-cpp-test-Ice-properties-run.py
>Only in /usr/ports/devel/ice/files: patch-cpp-test-IceGrid-admin-run.py
>Only in /usr/ports/devel/ice/files: patch-cpp-test-IceGrid-deployer-AllTests.cpp
>Only in /usr/ports/devel/ice/files: patch-cpp-test-IceGrid-deployer-Makefile
>Only in /usr/ports/devel/ice/files: patch-cpp-test-IceGrid-deployer-application.xml
>Only in /usr/ports/devel/ice/files: patch-cpp-test-IceGrid-distribution-AllTests.cpp
>Only in /usr/ports/devel/ice/files: patch-cpp-test-IceGrid-distribution-application.xml
>Only in /usr/ports/devel/ice/files: patch-cpp-test-IceGrid-distribution-run.py
>Only in /usr/ports/devel/ice/files: patch-cpp-test-IceGrid-session-run.py
>Only in /usr/ports/devel/ice/files: patch-cpp-test-IceSSL-configuration-AllTests.cpp
>Only in /usr/ports/devel/ice/files: patch-cpp-test-IceSSL-configuration-run.py
>Only in /usr/ports/devel/ice/files: patch-cpp-test-Slice-headers-run.py
>Only in /usr/ports/devel/ice/files: patch-cpp-test-Slice-unicodePaths-run.py
>Only in /usr/ports/devel/ice/files: patch-cpp-test-include-TestCommon.h
>Only in /usr/ports/devel/ice/files: patch-php-Makefile
>Only in /usr/ports/devel/ice/files: patch-php-config-Make.rules.php
>Only in /usr/ports/devel/ice/files: patch-php-lib-Makefile
>Only in /usr/ports/devel/ice/files: patch-python-Makefile
>Only in /usr/ports/devel/ice/files: patch-python-config-Make.rules
>Only in /usr/ports/devel/ice/files: patch-python-modules-IcePy-Types.cpp
>Only in /usr/ports/devel/ice/files: patch-python-modules-IcePy-Types.h
>Only in /usr/ports/devel/ice/files: patch-python-python-Makefile
>Only in /usr/ports/devel/ice/files: patch-python-test-Ice-info-AllTests.py
>Only in /usr/ports/devel/ice/files: patch-python-test-Ice-properties-run.py
>Only in /usr/ports/devel/ice/files: patch-python-test-Slice-unicodePaths-run.py
>Only in /usr/ports/devel/ice/files: patch-scripts-Expect.py
>Only in /usr/ports/devel/ice/files: patch-scripts-IceGridAdmin.py
>Only in /usr/ports/devel/ice/files: patch-scripts-TestUtil.py
>
>Note: The above was not expected. So I tried:
>
># ls -Tld /mnt/devel/ice/files/*
>-rw-r--r--  1 root  wheel   755 Apr 21 21:07:54 2021 /mnt/devel/ice/files/Make.rules.FreeBSD
>-rw-r--r--  1 root  wheel  1542 Apr 21 21:07:54 2021 /mnt/devel/ice/files/patch-config-Make.common.rules
>-rw-r--r--  1 root  wheel   388 Apr 21 21:07:54 2021 /mnt/devel/ice/files/patch-cpp-Makefile
>-rw-r--r--  1 root  wheel  1695 Apr 21 21:07:54 2021 /mnt/devel/ice/files/patch-cpp-allTests.py
>-rw-r--r--  1 root  wheel  1112 Apr 21 21:07:54 2021 /mnt/devel/ice/files/patch-cpp-config-Make.rules
>-rw-r--r--  1 root  wheel  1512 Apr 21 21:07:54 2021 /mnt/devel/ice/files/patch-cpp-include-Ice-FactoryTableInit.h
>-rw-r--r--  1 root  wheel  1496 Apr 21 21:07:54 2021 /mnt/devel/ice/files/patch-cpp-include-IceUtil-Config.h
>-rw-r--r--  1 root  wheel   447 Apr 21 21:07:54 2021 /mnt/devel/ice/files/patch-cpp-include-IceUtil-ScannerConfig.h
>-rw-r--r--  1 root  wheel   588 Apr 21 21:07:54 2021 /mnt/devel/ice/files/patch-cpp-src->Glacier2CryptPermissionsVerifier-CryptPermissionsVerifierI.cpp
>-rw-r--r--  1 root  wheel   384 Apr 21 21:07:54 2021 /mnt/devel/ice/files/patch-cpp-src-Ice-ProxyFactory.cpp
>-rw-r--r--  1 root  wheel   691 Apr 21 21:07:54 2021 /mnt/devel/ice/files/patch-cpp-src-IceGrid-PluginFacadeI.cpp
>-rw-r--r--  1 root  wheel   440 Apr 21 21:07:54 2021 /mnt/devel/ice/files/patch-cpp-src-IceGrid-RegistryI.cpp
>-rw-r--r--  1 root  wheel   543 Apr 21 21:07:54 2021 /mnt/devel/ice/files/patch-cpp-src-IceSSL-Makefile
>-rw-r--r--  1 root  wheel   990 Apr 21 21:07:54 2021 /mnt/devel/ice/files/patch-cpp-test-Glacier2-ssl-Server.cpp
>-rw-r--r--  1 root  wheel  1280 Apr 21 21:07:54 2021 /mnt/devel/ice/files/patch-cpp-test-Glacier2-staticFiltering-run.py
>-rw-r--r--  1 root  wheel   867 Apr 21 21:07:54 2021 /mnt/devel/ice/files/patch-cpp-test-Ice-info-AllTests.cpp
>-rw-r--r--  1 root  wheel  3337 Apr 21 21:07:54 2021 /mnt/devel/ice/files/patch-cpp-test-Ice-metrics-AllTests.cpp
>-rw-r--r--  1 root  wheel   752 Apr 21 21:07:54 2021 /mnt/devel/ice/files/patch-cpp-test-Ice-objects-Makefile
>-rw-r--r--  1 root  wheel   730 Apr 21 21:07:54 2021 /mnt/devel/ice/files/patch-cpp-test-Ice-properties-run.py
>-rw-r--r--  1 root  wheel   718 Apr 21 21:07:54 2021 /mnt/devel/ice/files/patch-cpp-test-IceGrid-admin-run.py
>-rw-r--r--  1 root  wheel  8800 Apr 21 21:07:54 2021 /mnt/devel/ice/files/patch-cpp-test-IceGrid-deployer-AllTests.cpp
>-rw-r--r--  1 root  wheel   438 Apr 21 21:07:54 2021 /mnt/devel/ice/files/patch-cpp-test-IceGrid-deployer-Makefile
>-rw-r--r--  1 root  wheel   787 Apr 21 21:07:54 2021 /mnt/devel/ice/files/patch-cpp-test-IceGrid-deployer-application.xml
>-rw-r--r--  1 root  wheel   997 Apr 21 21:07:54 2021 /mnt/devel/ice/files/patch-cpp-test-IceGrid-distribution-AllTests.cpp
>-rw-r--r--  1 root  wheel   434 Apr 21 21:07:54 2021 /mnt/devel/ice/files/patch-cpp-test-IceGrid-distribution-application.xml
>-rw-r--r--  1 root  wheel   459 Apr 21 21:07:54 2021 /mnt/devel/ice/files/patch-cpp-test-IceGrid-distribution-run.py
>-rw-r--r--  1 root  wheel   741 Apr 21 21:07:54 2021 /mnt/devel/ice/files/patch-cpp-test-IceGrid-session-run.py
>-rw-r--r--  1 root  wheel  1179 Apr 21 21:07:54 2021 /mnt/devel/ice/files/patch-cpp-test-IceSSL-configuration-AllTests.cpp
>-rw-r--r--  1 root  wheel  1038 Apr 21 21:07:54 2021 /mnt/devel/ice/files/patch-cpp-test-IceSSL-configuration-run.py
>-rw-r--r--  1 root  wheel  3490 Apr 21 21:07:54 2021 /mnt/devel/ice/files/patch-cpp-test-Slice-headers-run.py
>-rw-r--r--  1 root  wheel   484 Apr 21 21:07:54 2021 /mnt/devel/ice/files/patch-cpp-test-Slice-unicodePaths-run.py
>-rw-r--r--  1 root  wheel   608 Apr 21 21:07:54 2021 /mnt/devel/ice/files/patch-cpp-test-include-TestCommon.h
>-rw-r--r--  1 root  wheel   341 Apr 21 21:07:54 2021 /mnt/devel/ice/files/patch-php-Makefile
>-rw-r--r--  1 root  wheel   951 Apr 21 21:07:54 2021 /mnt/devel/ice/files/patch-php-config-Make.rules.php
>-rw-r--r--  1 root  wheel   434 Apr 21 21:07:54 2021 /mnt/devel/ice/files/patch-php-lib-Makefile
>-rw-r--r--  1 root  wheel   322 Apr 21 21:07:54 2021 /mnt/devel/ice/files/patch-python-Makefile
>-rw-r--r--  1 root  wheel  2629 Apr 21 21:07:54 2021 /mnt/devel/ice/files/patch-python-config-Make.rules
>-rw-r--r--  1 root  wheel  3394 Apr 21 21:07:54 2021 /mnt/devel/ice/files/patch-python-modules-IcePy-Types.cpp
>-rw-r--r--  1 root  wheel  5092 Apr 21 21:07:54 2021 /mnt/devel/ice/files/patch-python-modules-IcePy-Types.h
>-rw-r--r--  1 root  wheel   668 Apr 21 21:07:54 2021 /mnt/devel/ice/files/patch-python-python-Makefile
>-rw-r--r--  1 root  wheel  2640 Apr 21 21:07:54 2021 /mnt/devel/ice/files/patch-python-test-Ice-info-AllTests.py
>-rw-r--r--  1 root  wheel   736 Apr 21 21:07:54 2021 /mnt/devel/ice/files/patch-python-test-Ice-properties-run.py
>-rw-r--r--  1 root  wheel   490 Apr 21 21:07:54 2021 /mnt/devel/ice/files/patch-python-test-Slice-unicodePaths-run.py
>-rw-r--r--  1 root  wheel   381 Apr 21 21:07:54 2021 /mnt/devel/ice/files/patch-scripts-Expect.py
>-rw-r--r--  1 root  wheel   793 Apr 21 21:07:54 2021 /mnt/devel/ice/files/patch-scripts-IceGridAdmin.py
>-rw-r--r--  1 root  wheel  2588 Apr 21 21:07:54 2021 /mnt/devel/ice/files/patch-scripts-TestUtil.py
>
>Note: So that indicated that the files were there on the
>machine that /mnt references. So attempting the original
>diff -r again:
>
># diff -r /usr/ports/devel/ice/files /mnt/devel/ice/files | more
>#
>
>(Empty difference.)
>
>Note: So after the explicit "ls -Tld /mnt/devel/ice/files/*"
>the odd result of the diff -r no longer happened: no
>differences reported.
>
>
>
>For reference (both machines reported):
>
># uname -apKU
>FreeBSD CA72_16Gp_ZFS 13.0-RELEASE FreeBSD 13.0-RELEASE #0 releng/13.0-n244733-ea31abc261ff-dirty: Thu Apr 29 >21:53:20 PDT 2021     root at CA72_4c8G_ZFS:/usr/obj/BUILDs/13_0R-CA72-nodbg-clang/usr/13_0R->src/arm64.aarch64/sys/GENERIC-NODBG-CA72  arm64 aarch64 1300139 1300139
>
># ~/fbsd-based-on-what-commit.sh
>branch: releng/13.0
>merge-base: ea31abc261ffc01b6ff5671bffb15cf910a07f4b
>merge-base: CommitDate: 2021-04-09 00:14:30 +0000
>ea31abc261ff (HEAD -> releng/13.0, tag: release/13.0.0, freebsd/releng/13.0) 13.0: update to RELEASE
>n244733 (--first-parent --count for merge-base)
>
># uname -apKU
>FreeBSD CA72_4c8G_ZFS 13.0-RELEASE FreeBSD 13.0-RELEASE #0 releng/13.0-n244733-ea31abc261ff-dirty: Thu Apr 29 >21:53:20 PDT 2021     root at CA72_4c8G_ZFS:/usr/obj/BUILDs/13_0R-CA72-nodbg-clang/usr/13_0R->src/arm64.aarch64/sys/GENERIC-NODBG-CA72  arm64 aarch64 1300139 1300139
>
># ~/fbsd-based-on-what-commit.sh
>branch: releng/13.0
>merge-base: ea31abc261ffc01b6ff5671bffb15cf910a07f4b
>merge-base: CommitDate: 2021-04-09 00:14:30 +0000
>ea31abc261ff (HEAD -> releng/13.0, tag: release/13.0.0, freebsd/releng/13.0) 13.0: update to RELEASE
>n244733 (--first-parent --count for merge-base)
>
>From zfs list commands (one machine per line shown):
>
>zopt0/usr/ports                           2.13G   236G     2.13G  /usr/ports
>zroot/usr/ports                           2.13G   113G     2.13G  /usr/ports
>
>I've no clue if ZFS is important to the odditity
>or not.
>
>The original mount command was on CA72_16Gp_ZFS:
>
># mount -onoatime,soft 192.168.1.170:/usr/ports/ /mnt/
The likely explanation for this is your use of a "soft" mount.
- If the NFS server is slow to respond or there is a temporary network issue,
   the RPC request can time out and then the
   syscall can fail with EINT/ETIMEDOUT. Since almost nothing, including the
    readdir(3) libc functions expect syscalls to fail this way...
    Then the cached directory is messed up.
    Doing the "ls" read the directory again and fixed the problem.

Try to reproduce it for a mount without the "soft" option.
(If a mount point is hung, due to an unresponsive server "umount -N /mnt"
 can usually get rid of it.)
Personally, I thought "soft" was a bad idea when Sun introduced it in NFS in 1985
and I still feel that way.
--> If you can reproduce it without "soft" then I can't explain it.
      To be honest, the directory reading/caching code in the NFSv3 client
      hasn't changed significantly in literally decades, as far as I can remember.

One additional thing to note is that cached directory contents are invalidated
when the directory's ctime changes. I am not sure how/if/when ZFS changes a
directory's ctime. However, if it was badly broken, I'd hear about this a lot.
(If the ZFS change to ZoL has changed its ctime handling, that might also explain it
 and I'll be hearing a lot more soon as FreeBSD13 becomes adopted. I never use ZFS and,
 as such, never test with it.)
--> For UFS, if you use mtime, directory caching does not work as well, which is
       why the client directory caching code uses ctime and not mtime to detect that
       a directory has changed and cached directory blocks need to be invalidated.

Jason Bacon did report a directory reading issue some months ago that never
quite got resolved, although I recall he said he couldn't reproduce it after a
system update, so he thought it was related to some local change he had made.
(I can't remember his email or I'd add him to the cc so he could remind me what
 his case was. I do recall it being somewhat reproducible and happened for both
 UFS and ZFS.)
The network is just a local EtherNet.

===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)

_______________________________________________
freebsd-stable at freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"



More information about the freebsd-stable mailing list