zfs recv hangs in kmem arena

[James R. Van Artsdalen]

The zfs recv / kmem arena hang happens with -CURRENT as well as
10-STABLE, on two different systems, with 16GB or 32GB of RAM, from
memstick or normal multi-user environments,

Hangs usually seem to hapeen 1TB to 3TB in, but last night one run hung
after only 4.35MB.

On 9/26/2014 1:42 AM, James R. Van Artsdalen wrote:
> FreeBSD BLACKIE.housenet.jrv 10.1-BETA2 FreeBSD 10.1-BETA2 #2 r272070M:
> Wed Sep 24 17:36:56 CDT 2014    
> james at BLACKIE.housenet.jrv:/usr/obj/usr/src/sys/GENERIC  amd64
>
> With current STABLE10 I am unable to replicate a ZFS pool using zfs
> send/recv without zfs hanging in state "kmem arena", within the first
> 4TB or so (of a 23TB Pool).
>
> The most recent attempt used this command line
>
> SUPERTEX:/root# zfs send -R BIGTEX/UNIX at syssnap | ssh BLACKIE zfs recv
> -duvF BIGTOX
>
> though local replications fail in kmem arena too.
>
> The two machines I've been attempting this on have 16BG and 32GB of RAM
> each and are otherwise idle.
>
> Any suggestions on how to get around, or investigate, "kmem arena"?
>
> # top
> last pid:  3272;  load averages:  0.22,  0.22,  0.23                  up
> 0+08:25:02  01:32:07
> 34 processes:  1 running, 33 sleeping
> CPU:  0.0% user,  0.0% nice,  0.1% system,  0.0% interrupt, 99.9% idle
> Mem: 21M Active, 82M Inact, 15G Wired, 28M Cache, 450M Free
> ARC: 12G Total, 24M MFU, 12G MRU, 23M Anon, 216M Header, 47M Other
> Swap: 16G Total, 16G Free
>
>   PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU
> COMMAND
>  1173 root          1  52    0 86476K  7780K select  0 124:33   0.00% sshd
>  1176 root          1  46    0 87276K 47732K kmem a  3  48:36   0.00% zfs
>   968 root         32  20    0 12344K  1888K rpcsvc  0   0:13   0.00% nfsd
>  1009 root          1  20    0 25452K  2864K select  3   0:01   0.00% ntpd
> ...