slowdown of zfs (tx->tx)

[Andriy Gapon]

on 16/01/2013 12:14 Steven Hartland said the following:
> ----- Original Message ----- From: "Nicolas Rachinsky"
> 
>> * Andriy Gapon <avg at FreeBSD.org> [2013-01-16 11:39 +0200]:
>>> on 16/01/2013 02:16 Artem Belevich said the following:
>>> > It appears that lots of threads are stuck in
>>> > metaslab_activate->space_map_load_wait path.
>>>
>>> another thing to check - is your pool nearly full.
>>
>> Don't think so:
>> NAME                                 USED  AVAIL  REFER  MOUNTPOINT
>> pool1                               5.52T   697G  11.9M  /pool1
> 
> You only have ~11% free so yer it is pretty full ;-)

Nicolas,

just in case, Steve is not kidding.

Those free hundreds of gigabytes could be spread over the terabytes and could be
quite fragmented if the pool has a history of adding and removing lots of files.
 ZFS could be spending quite a lot of time in that case when it looks for some
free space and tries to minimize further fragmentation.

Empirical/anecdotal safe limit on pool utilization is said to be about 70-80%.

You can test if this guess is true by doing the following:
kgdb -w
(kgdb) set metaslab_min_alloc_size=4096

If performance noticeably improves after that, then this is your problem indeed.

-- 
Andriy Gapon