Re: ZFS operations hanging, but no visible errors?

Chris Ross <cross+freebsd@distal.com> wrote on 2021-11-05:

> Hey there.  I have a server running FreeBSD 13.0-RELEASE,
> with a large ZFS zpool.  I have a UFS mirror on hardware
> raid, then a bunch of JBOD disks in a pool.  I recently
> added a new vdev to this pool, which may or may not be related.
> 
> Today, I started an rsync of a large (100GB) file from
> the pool to another host.  After a while (7%), it seemed
> no progres was being made.  I tried to kill the rsync,
> which didn’t exit, or suspend.  Now anything that touches
> the pool seems to hang.  But, the system is otherwise
> functional, console shows no issues, the controller
> (via out-of-band management interface) shows all disks
> as having no errors or issues.
> 
> Any idea what I should be looking for, and if there’s
> any way to recover it without reboot?

You can check with "procstat -kk" where the hanging
processes are stuck.

The output of:
dtrace -qn 'zfs-dbgmsg {printf("%Y: %s\n", walltimestamp, stringof(arg0))}'
could be useful as well.

Fabian