zfs recv went idle after 20+ hours

[Martin Simmons]

>>>>> On Fri, 18 Aug 2017 23:30:49 -0400, Chris Ross said:
> 
>   From earlier posts on this list, many know I was dealing with an issue on a zfs pool.  I eventually got it running in an off way, and backed it up (via zfs send to a file in UFS).  I then destroyed and recreated the pool as I wanted it.  For the last 23 hours or so, I’ve been running a “zfs recv” from the large send output (~5.2T).  It was cooking along, more slowly than I’d expected, but moving.
> 
>   I looked in on it a short while ago, and the pv I’m using on the input file was reporting 0.0 B/s, 77% complete.  Looking at the iostat I had running, I could see the raidz disks still seeing activity, but less than a minute later that stopped.  Now, all disks on the system seem idle.  The “zfs recv” process is idle according to ps (STAT I+), and a ktrace on the zfs process shows no activity either.  The window in which I was running the commands look as follows:
> 
> cross at hyrule[~](511): zpool list
> NAME    SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
> tank   18.8T   949K  18.8T         -     0%     0%  1.00x  ONLINE  -
> zroot  50.5G  34.7G  15.8G         -    55%    68%  1.00x  ONLINE  -
> cross at hyrule[~](512): pv /mnt/tank at 20170815| sudo zfs recv -Fvu tank
> receiving full stream of tank at 20170815 into tank at 20170815
> received 46.3KB stream in 1 seconds (46.3KB/sec)
> receiving full stream of tank/deluge at 20170815 into tank/deluge at 20170815
> received 2.87TB stream in 45452 seconds (66.3MB/sec)          ] 55% ETA 10:18:14
> receiving full stream of tank/timecapsule at 20170815 into tank/timecapsule at 20170815
> 3.99TiB 22:54:18 [0.00 B/s] [========================>         ] 76% ETA 7:04:49
> 
>   The time-of-day and ETA value keep increasing, but other than that, it’s just sitting.  zfs list shows:
> 
> hyrule# zfs list -t all | grep tank
> tank                  3.89T  8.28T  73.3K  none
> tank at 20170815             0      -  73.3K  -
> tank/deluge           2.86T  8.28T  2.86T  /data/deluge
> tank/deluge at 20170815      0      -  2.86T  -
> tank/timecapsule      1024G  8.28T  1024G  /data/timecapsule
> 
>   I’m going to leave it this way for a while, but it’s already been about 10 minutes since it started, so I don’t expect that it will free itself.  Does anyone have a suggestion, or something I should look at?  Did I do something wrong?

Check that top and/or ps show nothing else using CPU.

You could use procstat -kk to see what each unexpectedly idle process is
waiting for in the kernel.

__Martin