ZFS performance gains real or imaginary?

[Matt Simerson]

On Dec 24, 2008, at 9:29 PM, Adam McDougall wrote:

>> On Wed, Dec 24, 2008 at 01:00:14PM -0800, Matt Simerson wrote:
>>
>>  On Dec 23, 2008, at 7:48 PM, Adam McDougall wrote:
>>
>>>> On Tue, Dec 23, 2008 at 12:43:47PM -0800, Matt Simerson wrote:
>>>>
>>>>> On Dec 18, 2008, at 4:12 PM, Andrew Snow wrote:
>>>>>
>>>>>> If so, then I really should be upgrading my production ZFS  
>>>>>> servers
>>>>>> to the latest -HEAD.
>>>>>
>>>>> Thats correct, that is the only way to get the best working  
>>>>> version
>>>>> of ZFS.  Of course, then everything is unstable and broken - eg.
>>>>> SMBFS became unusable for me and would crash the server.
>>>>
>>>> Unfortunately, the newer kernel hangs much more frequently.
>>>>
>>>> I have these settings in /boot/loader.conf
>>>>
>>>> 	vm.kmem_size="1536M"
>>>> 	vm.kmem_size_max="1536M"
>>>> 	vfs.zfs.arc_max="100M"
>>>>
>>>> I have also experimented with vfs.zfs.prefetch_disable,
>>>> vfs.zfs.arc_min in the past, and I'm open to suggestions on what
>>>> might help under this workload (multiple concurrent rsync
>>>> processes from remote systems to this one).
>>>
>>> Can you try:
>>>
>>> vm.kmem_size=2G
>>> vm.kmem_size_max=2G
>>> vfs.zfs.arc_max=512M
>>>
>>> This has been working for me on one amd64 system that only
>>> has 2G of  ram but had similar problem frequency to yours.  I
>>> don't know if its  coincidence with the data that I am rsyncing
>>> lately, but: 10:47PM  up 22 days,  7:12
>>
>>  I made it 23 minutes. I've reduced my rsync concurrency to 1, so I'm
>>  not hitting the system nearly as hard but it seems not to matter.
>>
>>  Other workloads, like a 'make buildworld' will complete with no
>>  problems. For whatever reason, rsync sessions of entire unix systems
>>  to my backup servers are very troublesome.
>>
>>  Matt
>
> Ok.  Since you have 16G of ram, I suppose you could try setting both  
> kmem
> sizes to something like 8G to see if it makes a difference?  I'm  
> getting
> a feeling that even if we don't see an outright failure, it might be
> deadlocking due to a kmem shortage.


back01# w
10:17PM  up 40 mins, 2 users, load averages: 4.20, 3.07, 1.74

This is with:

    vm.kmem_size="4G"
    vm.kmem_size_max="4G"
    vfs.zfs.arc_max="512M"

I'll let it trundle along with that setting and see how long it lasts.

Matt

PS: These settings earlier today resulted in 12+ hours of uptime,  
until I rebooted to test raising kmem_size to 4G.

    vm.kmem_size="2G"
    vm.kmem_size_max="2G"
    vfs.zfs.arc_max="512M"
    vfs.zfs.zil_disable=1
    vfs.zfs.prefetch_disable=1

PPS: If/when it hangs with 4G, I'll raise it again to 6 or 8 GB and  
see how long it lasts. Whatever pattern emerges might be useful for  
Pawel.