zfs -> ufs rsync: livelock in wdrain state

[Konstantin Belousov]

On Tue, Jan 08, 2013 at 12:19:15AM +0400, Dmitry Morozovsky wrote:
> Dear colleagues,
> 
> I have archive server with pretty large ZFS (24*2T in single raidz2 raidgroup)
> 
> Sometimes we moved really old archives to external SATA drives, which are 
> formatted with UFS2/SU.  Files are copied via rsync
> 
> The system in question is stable/8; upgrade to stable/9 is planned, but not yet 
> completed.
> 
> Now, during last rsync, the process is stuck as
> 
> dump.2012062219.bin.gz
>   3208015437 100%  102.42MB/s    0:00:29 (xfer#66, to-check=196/721)
> dump.2012062220.bin.gz
> load: 0.01  cmd: rsync 47543 [wdrain] 1904.69r 443.01u 241.12s 0% 1736k
> ^C
> rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(645) 
> [sender=3.0.9]
> 
> As we can see, rsync writer stops in wdrain state.
> 
> I terminated it by ^C in terminal session, as it was not autogenerated 
> backup.
> 
> Now, zfs and other system is working seemingly well, but trying to sync 
> manually stucks console forever:
> 
> root at moose:/ar# sync
> load: 0.00  cmd: sync 67229 [wdrain] 468.17r 0.00u 0.00s 0% 596k
> 
> Any hints? Quick searching throug freebsd mailing lists and/or open PRs does 
> not reveal much.
> 

Are there any kernel messages about the disk system ?

The wdrain means that the amount of the dirty buffers accumulated exceeds
the allowed maximum. The transient 'wdrain' state is normal on a machine
doing lot of writes to a filesystem using buffer cache, say UFS. Failure
to clean the dirty buffers is usually related to the disk i/o stalling.

It cannot be denied that a bug could cause stuck 'wdrain' state, but
in the last five or so years all the cases I investigated were due to
disks.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 834 bytes
Desc: not available
URL: <http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20130108/04979639/attachment.sig>