Disk/ZFS activity crash on 11.2-STABLE

[Mike Tancsa]

On 7/12/2018 2:35 PM, Jim Long wrote:
> 
> Yes, ARC usage is high.  top shows:
> 
> last pid:  2215;  load averages:  1.50,  1.69,  1.69   up 0+00:44:59  11:02:00
> 27 processes:  2 running, 25 sleeping
> CPU:  0.0% user,  0.0% nice,  9.0% system,  0.1% interrupt, 90.9% idle
> Mem: 1416K Active, 11M Inact, 2788K Laundry, 91G Wired, 1236M Free
> ARC: 84G Total, 68M MFU, 83G MRU, 15M Anon, 877M Header, 4078K Other
>      83G Compressed, 148G Uncompressed, 1.78:1 Ratio
> Swap: 16G Total, 15M Used, 16G Free
> 
> I'm re-running the test now.  I had shut off hyperthreading to go down to
> 24 cores, but it's back on now.  This server will be a computational resource,
> not just storage, so I'd like to find a way to stablize the configuration with
> hyperthreading enabled, if possible.
I would leave HT on.  Based on the zfs stats

CACHE HITS BY DATA TYPE:
	  Demand Data:			51.38%	457.85k
	  Prefetch Data:		0.00%	2
	  Demand Metadata:		44.71%	398.41k
	  Prefetch Metadata:		3.91%	34.86k

	CACHE MISSES BY DATA TYPE:
	  Demand Data:			3.22%	17.71k
	  Prefetch Data:		81.77%	449.51k
	  Demand Metadata:		8.57%	47.14k
	  Prefetch Metadata:		6.43%	35.35k


If I read this right, you are not hurting too much for meta data misses.
I find things like
zfs list -t snapshots
REALLY slow down if there is not enough caching for meta data where
there are a lot of files and directories.

eg. on a box that has several million small files and many many
directories, I increased the amount of RAM to vfs.zfs.arc_meta_limit and
it helped a LOT. Even then I still see a lot of misses (it was worse before)

        CACHE HITS BY DATA TYPE:
          Demand Data:                  10.82%  5.58b
          Prefetch Data:                0.66%   339.34m
          Demand Metadata:              50.83%  26.22b
          Prefetch Metadata:            37.69%  19.44b

        CACHE MISSES BY DATA TYPE:
          Demand Data:                  6.79%   680.20m
          Prefetch Data:                8.43%   844.48m
          Demand Metadata:              79.52%  7.97b
          Prefetch Metadata:            5.26%   527.45m

But I dont think thats your case.  I would try and set a ceiling. On
RELENG_11 you dont need to reboot

Try
sysctl -w vfs.zfs.arc_max=77946198016

which shaves off 20G from what ARC can gobble up. Not sure if thats your
issue, but it is an issue for some users.

If you are still hurting for caching, an SSD drive or NVME and make it a
caching device for your pool.

and what does
zpool status
show ?

	---Mike


-- 
-------------------
Mike Tancsa, tel +1 519 651 3400 x203
Sentex Communications, mike at sentex.net
Providing Internet services since 1994 www.sentex.net
Cambridge, Ontario Canada