Slow reboots due to ZFS cleanup in kern_shutdown() .. zio_fini()

[Mark Johnston]

On Mon, Dec 02, 2019 at 11:39:01PM +0100, Peter Eriksson wrote:
> Sigh.
> 
> Slight correction, the output below should have said uma_zdestroy() and not uma_zfree_arg() (wrong printf text, but the right times).
> 
> After an uptime of 7 hours, a reboot have these times (I removed the “uma” printf in this run):
> 
> kmem_cache_destroy(zio_data_buf_cache[8]) took 2 seconds
> kmem_cache_destroy(zio_buf_cache[10]) took 6 seconds
> kmem_cache_destroy(zio_buf_cache[14]) took 2 seconds
> kmem_cache_destroy(zio_buf_cache[16]) took 136 seconds
> kmem_cache_destroy(zio_buf_cache[20]) took 31 seconds
> kmem_cache_destroy(zio_buf_cache[28]) took 303 seconds
> kmem_cache_destroy(zio_buf_cache[224]) took 89 seconds
> kmem_cache_destroy(zio_data_buf_cache[224]) took 31 seconds
> 
> This is on a mostly idle server (well, apart from compiling the kernel code :-) and some snapshots being taken of all filesystems (once per hour).
> 
> 
> So now on to finding out why uma_destroy() is taking so long… :-).

uma_destroy() frees all of the memory cached in the zone back to the
page allocator.  This operation takes time proportional to the number of
cached items.  I would expect most of the time to be spent in
zone_reclaim(), called by zone_dtor().

Before rebooting, could you try grabbing the output of vmstat -z?