Slow reboots due to ZFS cleanup in kern_shutdown() .. zio_fini()

[Mark Johnston]

On Tue, Dec 03, 2019 at 11:03:11AM +0200, Andriy Gapon wrote:
> On 03/12/2019 00:54, Mark Johnston wrote:
> > On Mon, Dec 02, 2019 at 11:39:01PM +0100, Peter Eriksson wrote:
> >> Sigh.
> >>
> >> Slight correction, the output below should have said uma_zdestroy() and not uma_zfree_arg() (wrong printf text, but the right times).
> >>
> >> After an uptime of 7 hours, a reboot have these times (I removed the “uma” printf in this run):
> >>
> >> kmem_cache_destroy(zio_data_buf_cache[8]) took 2 seconds
> >> kmem_cache_destroy(zio_buf_cache[10]) took 6 seconds
> >> kmem_cache_destroy(zio_buf_cache[14]) took 2 seconds
> >> kmem_cache_destroy(zio_buf_cache[16]) took 136 seconds
> >> kmem_cache_destroy(zio_buf_cache[20]) took 31 seconds
> >> kmem_cache_destroy(zio_buf_cache[28]) took 303 seconds
> >> kmem_cache_destroy(zio_buf_cache[224]) took 89 seconds
> >> kmem_cache_destroy(zio_data_buf_cache[224]) took 31 seconds
> >>
> >> This is on a mostly idle server (well, apart from compiling the kernel code :-) and some snapshots being taken of all filesystems (once per hour).
> >>
> >>
> >> So now on to finding out why uma_destroy() is taking so long… :-).
> > 
> > uma_destroy() frees all of the memory cached in the zone back to the
> > page allocator.  This operation takes time proportional to the number of
> > cached items.  I would expect most of the time to be spent in
> > zone_reclaim(), called by zone_dtor().
> 
> But spending *minutes* there is really unexpected.
> I have never seen anything like that.
> I wonder if there is anything untypical about the system's hardware (like a very
> big number of processors) or configuration.

Indeed, looking at the vmstat -z output this doesn't really make sense.
It would be useful to confirm that we are in fact spending most of the
time reclaiming items.