ZFS server has gone crazy slow

[Peter Eriksson]

Sure You can find my stuff at my Github page:

  https://github.com/ptrrkssn/freebsd-stuff <https://github.com/ptrrkssn/freebsd-stuff>


The things needed are the /etc/rc.d scripts (zfs, nfsd, samba_server, cold) scripts in “rc.d” and the “zfs-mountall” in “scripts” that goes into /sbin.

What they basically do are use a helper function “waitfor” to wait for a “${name}.done” file to appear in /var/run and then just 

   (waitfor zfs ; run_rc_command “$1”) &

Instead of just the “run_rc_command “$1”. And then we have the scripts create those “${name}.done” files when they have started their daemons.
Not foolproof (should check that the daemon really is successfully running and more before signalling “done”).


scripts/zfs-mountall 

Mounts all filesystems under a specific DATASET sequentially. Basically what a “zfs mount -r” would have done if that option had existed.


rc.d/zfs

Executes /sbin/zfs-mountall to make sure all filesystems in the ‘zroot’ (or whatever dataset / is mounted from) is mounted first. This is needed to make sure all important filesystems are mounted (like various /var filesystems etc). Then basically does:

    (
        rm -f /var/run/zfs*.done
        zfs mount -va >/var/log/zfs/mount.log && touch /var/run/zfs-mount.done
        zfs share -a  >/var/log/zfs/share.log && touch /var/run/zfs.done
   ) &


rc.d/mountd, rc.d/nfsd, rc.d/ctld & rc.d/samba_server all contains a function:

  waitfor() {
    FIRST=yes
    while [ ! -f "/var/run/$1.done" ]; do
        if [ $FIRST = yes ]; then
            echo "${name}: Waiting for '$1' to finish..."
            FIRST=no
        fi
        sleep 1
    done
  }

It probably should go into /etc/rc.subr or somewhere else but… “waitfor” basically is loop and waits for the “${name}.done” file to appear in /var/run.


rc.d/mountd then ends with this instead of just: ‘ run_rc_command “$1” ‘

  case "$1" in
     faststart)
        rm -f "/var/run/${name}.done"
        (waitfor zfs ; run_rc_command "$1" ; touch "/var/run/${name}.done") &
        ;;
      *)
        run_rc_command "$1"
        ;;
  esac


rc.d/nfsd ends with:
  case "$1" in
    faststart)
        (waitfor "mountd" ; run_rc_command "$1") &
        ;;
    *)
        run_rc_command "$1"
        ;;
  esac

(And similar for cold)

And rc.d/samba_server has a bit more modifications (we only want “smbd” to delay it’s startup, not winbindd or numbed) so does this in the middle of the script. Also just waits for the “zfs mount -a” to be finished, doesn’t have to wait for “zfs share -a” (NFS)):

        # Daemon should be enabled and running                                                           
        if ( [ -n "${rcvar}" ] && checkyesno "${rcvar}" ) || [ -n "$force_run" ]; then
            if [ "${rc_arg}" = "start" -a "${name}" = "smbd" ]; then
                (waitfor "zfs-mount" ; run_rc_command "${_rc_prefix}${rc_arg}" ${rc_extra_args}) &
            else
                run_rc_command "${_rc_prefix}${rc_arg}" ${rc_extra_args}
		# If any of the commands failed, take it as a global result                              
                result=$((${result} || $?))
            fi
	fi
 
- Peter



> On 12 Apr 2020, at 09:17, Ireneusz Pluta/wp.pl <ipluta at wp.pl> wrote:
> 
> W dniu 2020-04-11 o 23:24, Peter Eriksson pisze:
>> (We have modified our rc.d startup scripts so we do the “zfs mount -a” part in the background so the rest of the system doesn’t have to wait for it (so we can get a ssh login prompt and login and check the progress of the filesystem mounts. (And then “nfs exports”, nfsd and samba also waits in the background for that to finish before starting up. A bit of a hack but it works (a real parallel server manager with dependencies would have been better
> would you share these modifications?
> 
>> :-)