[Bug 244665] Very slow NFS I/O during ZFS resilver - _sleep() sleeps really long...

[bugzilla-noreply at freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=244665

            Bug ID: 244665
           Summary: Very slow NFS I/O during ZFS resilver - _sleep()
                    sleeps really long...
           Product: Base System
           Version: 11.3-RELEASE
          Hardware: Any
                OS: Any
            Status: New
          Severity: Affects Some People
          Priority: ---
         Component: kern
          Assignee: bugs at FreeBSD.org
          Reporter: pen at lysator.liu.se

I've been investigating an issue we're having that gives our users a really
lousy NFS experience during ZFS resilver on one of our servers. 

In order to try to pinpoint exactly what's going on I've tried various things.
Lately I've been dtrace the call path for "mkdir()" and it seems that it is the
_sleep() kernel call that sometimes sleeps for _really_ long times like 1-20
seconds...

Sure, the kernel is busy doing the ZFS resilver and other stuff but this is
silly... Any ideas? Too low priority of the _sleep (prio=99?) compared to other
kernel threads so it never gets scheduled?


One example:

 21  54842               nfsrvd_dorpc:entry Start
 21  47273                 nfsv4_lock:entry Start(lp->nfslock_lock=6,
iwantlock=0)
 21  37590                  nfsmsleep:entry Start(ffffffff81e9982c,
ffffffff81e998a0, 99)
 21  54396                     _sleep:entry Start(prio=99, timeo=0)
 16  54397                    _sleep:return   10942618 µs
 16  37591                 nfsmsleep:return   10942626 µs
 16  37590                  nfsmsleep:entry Start(ffffffff81e9982c,
ffffffff81e998a0, 99)
 16  54396                     _sleep:entry Start(prio=99, timeo=0)
  7  54397                    _sleep:return        344 µs
  7  37591                 nfsmsleep:return        352 µs
  7  47274                nfsv4_lock:return   10942987 µs
  7  13973           nfsrvd_statstart:entry Start
  7  13974          nfsrvd_statstart:return          2 µs
  7  44369     nfsrv_mallocmget_limit:entry Start
  7  44370    nfsrv_mallocmget_limit:return          2 µs
  7  13971             nfsrvd_statend:entry Start
  7  13972            nfsrvd_statend:return          2 µs
  7  13973           nfsrvd_statstart:entry Start
  7  13974          nfsrvd_statstart:return          1 µs
  7  51444                nfsrv_mtofh:entry Start
  7  51445               nfsrv_mtofh:return          2 µs
  7  47329                nfsd_fhtovp:entry Start
  7  48059           __mtx_lock_flags:entry Start
  7  48060          __mtx_lock_flags:return          2 µs
  7  47330               nfsd_fhtovp:return         14 µs
  7  13971             nfsrvd_statend:entry Start
  7  13972            nfsrvd_statend:return          1 µs
  7  13973           nfsrvd_statstart:entry Start
  7  13974          nfsrvd_statstart:return          1 µs
  7  43887             nfsrvd_getattr:entry Start
  7  38608      nfsv4root_getreferral:entry Start
  7  38609     nfsv4root_getreferral:return          2 µs
  7  43888            nfsrvd_getattr:return         22 µs
  7  13971             nfsrvd_statend:entry Start
  7  13972            nfsrvd_statend:return          1 µs
  7  54843              nfsrvd_dorpc:return   10943076 µs

-- 
You are receiving this mail because:
You are the assignee for the bug.