[Bug 244465] ZFS recursive snapshot with refquota-full filesystems causes DoS for NFS users

[bugzilla-noreply at freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=244465

            Bug ID: 244465
           Summary: ZFS recursive snapshot with refquota-full filesystems
                    causes DoS for NFS users
           Product: Base System
           Version: 11.3-RELEASE
          Hardware: Any
                OS: Any
            Status: New
          Severity: Affects Some People
          Priority: ---
         Component: misc
          Assignee: bugs at FreeBSD.org
          Reporter: pen at lysator.liu.se

This is probably related to the ZFS feature of throttling down the transaction
size when writing data to very near-full (refquota) filesystems but anyway. 

System:
Servers with many ZFS filesystems (HOME directories) for many users, shared via
NFSv4 and SMB (Samba).

We are seeing DoS issues for NFS users on the same machine where some _other_
user have filled their HOME directories so they are at, or very near (MB's),
their refquota limits at the time when we are taking our hourly recursive
snapshots of all filesystems on those servers.

   ("zfs snapshot -r DATA/homes" basically)

The NFS users (they have their HOME directories mounted from those servers) at
those times are seeing freezes for over a minute where the NFS server basically
seems to be unresponsive. We are also seeing NFS mount requests beeing denied
(or rather timed out) at the same times.

We haven't seen the same type of bug reports from our SMB users, but their
usage patterns are a bit different so this might just be that they haven't
noticed the same issue.

Possible workarounds (not tested yet):

1. Don't use "zfs snapshot -r" but instead manually loop over all filesystems
and skip taking snapshots of the ones that are nearly full. (Modify the source
for the "zfs" command to have an option to skip nearly-full filesystems when
doing a recursive snap)

2. Temporarily increase the refquota of problematic filesystems before running
"zfs snapshot -r". (Problem: We might not be able to lower the quota
afterwards).

3. Modify the zfs snapshot stuff in the kernel to ignore the quotas (since it's
run by the root users I think if might be reasonable, but possibly not so easy
to implement).

In general this slowing down of everything ZFS-related when a filesystem is
nearly full is starting to _really_ become a pain in the...


3.

-- 
You are receiving this mail because:
You are the assignee for the bug.