zfs recv went idle after 20+ hours

[Chris Ross]

  From earlier posts on this list, many know I was dealing with an issue on a zfs pool.  I eventually got it running in an off way, and backed it up (via zfs send to a file in UFS).  I then destroyed and recreated the pool as I wanted it.  For the last 23 hours or so, I’ve been running a “zfs recv” from the large send output (~5.2T).  It was cooking along, more slowly than I’d expected, but moving.

  I looked in on it a short while ago, and the pv I’m using on the input file was reporting 0.0 B/s, 77% complete.  Looking at the iostat I had running, I could see the raidz disks still seeing activity, but less than a minute later that stopped.  Now, all disks on the system seem idle.  The “zfs recv” process is idle according to ps (STAT I+), and a ktrace on the zfs process shows no activity either.  The window in which I was running the commands look as follows:

cross at hyrule[~](511): zpool list
NAME    SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
tank   18.8T   949K  18.8T         -     0%     0%  1.00x  ONLINE  -
zroot  50.5G  34.7G  15.8G         -    55%    68%  1.00x  ONLINE  -
cross at hyrule[~](512): pv /mnt/tank at 20170815| sudo zfs recv -Fvu tank
receiving full stream of tank at 20170815 into tank at 20170815
received 46.3KB stream in 1 seconds (46.3KB/sec)
receiving full stream of tank/deluge at 20170815 into tank/deluge at 20170815
received 2.87TB stream in 45452 seconds (66.3MB/sec)          ] 55% ETA 10:18:14
receiving full stream of tank/timecapsule at 20170815 into tank/timecapsule at 20170815
3.99TiB 22:54:18 [0.00 B/s] [========================>         ] 76% ETA 7:04:49

  The time-of-day and ETA value keep increasing, but other than that, it’s just sitting.  zfs list shows:

hyrule# zfs list -t all | grep tank
tank                  3.89T  8.28T  73.3K  none
tank at 20170815             0      -  73.3K  -
tank/deluge           2.86T  8.28T  2.86T  /data/deluge
tank/deluge at 20170815      0      -  2.86T  -
tank/timecapsule      1024G  8.28T  1024G  /data/timecapsule

  I’m going to leave it this way for a while, but it’s already been about 10 minutes since it started, so I don’t expect that it will free itself.  Does anyone have a suggestion, or something I should look at?  Did I do something wrong?

  Thanks…

              - Chris

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20170818/e6715713/attachment.sig>