Slow reboots due to ZFS cleanup in kern_shutdown() .. zio_fini()

[Andriy Gapon]

On 03/12/2019 00:54, Mark Johnston wrote:
> On Mon, Dec 02, 2019 at 11:39:01PM +0100, Peter Eriksson wrote:
>> Sigh.
>>
>> Slight correction, the output below should have said uma_zdestroy() and not uma_zfree_arg() (wrong printf text, but the right times).
>>
>> After an uptime of 7 hours, a reboot have these times (I removed the “uma” printf in this run):
>>
>> kmem_cache_destroy(zio_data_buf_cache[8]) took 2 seconds
>> kmem_cache_destroy(zio_buf_cache[10]) took 6 seconds
>> kmem_cache_destroy(zio_buf_cache[14]) took 2 seconds
>> kmem_cache_destroy(zio_buf_cache[16]) took 136 seconds
>> kmem_cache_destroy(zio_buf_cache[20]) took 31 seconds
>> kmem_cache_destroy(zio_buf_cache[28]) took 303 seconds
>> kmem_cache_destroy(zio_buf_cache[224]) took 89 seconds
>> kmem_cache_destroy(zio_data_buf_cache[224]) took 31 seconds
>>
>> This is on a mostly idle server (well, apart from compiling the kernel code :-) and some snapshots being taken of all filesystems (once per hour).
>>
>>
>> So now on to finding out why uma_destroy() is taking so long… :-).
> 
> uma_destroy() frees all of the memory cached in the zone back to the
> page allocator.  This operation takes time proportional to the number of
> cached items.  I would expect most of the time to be spent in
> zone_reclaim(), called by zone_dtor().

But spending *minutes* there is really unexpected.
I have never seen anything like that.
I wonder if there is anything untypical about the system's hardware (like a very
big number of processors) or configuration.

-- 
Andriy Gapon