system is unresponsive and the amount of wired memory is cycling - zfs/iscsi ?

[Eugene M. Zheganin]

Hi.

I'm using a FreeBSD 11.0-R server as a SAN system (with a native iSCSI 
target). It has 12 disks attached via external enclosure and a Megaraid 
SAS 3003 mrsas(4) controller. Actually I'm using several FreeBSD in 
similar configurations as SAN systems, but this one is frequently 
unresponsive. This often looks like top/gstat/dmesg are locked on start 
and waiting for something, then, after several minutes, everything is 
back to normal. In the same time ls and zpool list are working fine, but 
zfs clone is not. In the dmesg I got tonns if messages like

ctl_datamove: tag 0x1086bc on (16:34:0) aborted
ctl_datamove: tag 0x1086a8 on (16:34:0) aborted
ctl_datamove: tag 0x10ec50 on (7:34:0) aborted
ctl_datamove: tag 0xf1cd6 on (13:34:0) aborted
ctl_datamove: tag 0xfcc88 on (15:34:0) aborted
ctl_datamove: tag 0xd3ed1 on (21:34:0) aborted
ctl_datamove: tag 0x1056f8 on (1:34:0) aborted


Not sure if they are related to the lock issue I'm having, so I thought 
I will just mention them. It seems like the system is starving on some 
resources, but I really have no idea about what could it be. When this 
lock is happening, a launched top stops to update it's output, so I 
don't know what is happening, I let the system run with default zfs 
settings, top show that the ARC has eaten out all of the memory. One 
more importang thing - just befor top stops refereshing the screen it 
shows that ARC has eaten all the memory, then, after lock is gone, top 
shows the server has 11G free memory back, so this happens in cycles and 
definitely has something to do with the amount of the wired memory.


Should I tune the zfs subsystem in some way ? If yes - what exactly 
should I tune ? I'm also running this system with a CTL patch increasing 
the target limit to 1024, may be this is eating some additional kernel 
memory, don't know if this is the source of this locking issue. I'm 
running other systems with the same patch, but they are running smoothly.


Thanks.

Eugene.