Slow reboots due to ZFS cleanup in kern_shutdown() .. zio_fini()

[Peter Eriksson]

We run without swap on our servers…

I’m trying to instrument the uma_zdestroy() code now, should provide some more details in an hour or two… :-)

- Peter


> On 3 Dec 2019, at 17:59, Bengt Ahlgren <bengt.ahlgren at ri.se> wrote:
> 
> Peter Eriksson <pen at lysator.liu.se <mailto:pen at lysator.liu.se>> writes:
> 
>>>> uma_destroy() frees all of the memory cached in the zone back to the
>>>> page allocator.  This operation takes time proportional to the number of
>>>> cached items.  I would expect most of the time to be spent in
>>>> zone_reclaim(), called by zone_dtor().
>>> 
>>> But spending *minutes* there is really unexpected.
>>> I have never seen anything like that.
>>> I wonder if there is anything untypical about the system's hardware
>>> (like a very
>>> big number of processors) or configuration.
>>> 
>>> -- 
>>> Andriy Gapon
>> 
>> It’s a fairly standard Dell PowerEdge R730xd server with Intel Xeon
>> E5-2620v4 CPUs & 256GB of RAM… (and an LSI SAS3 HBA and Intel 10GE
>> ethernet)
>> 
>> CPU: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz (2100.04-MHz K8-class CPU)
>>  Origin="GenuineIntel"  Id=0x406f1  Family=0x6  Model=0x4f  Stepping=1
>>  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
>>  Features2=0x7ffefbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
>>  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
>>  AMD Features2=0x121<LAHF,ABM,Prefetch>
>>  Structured Extended
>> Features=0x21cbfbb<FSGSBASE,TSCADJ,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,NFPUSG,PQE,RDSEED,ADX,SMAP,PROCTRACE>
>>  Structured Extended Features3=0x9c000400<MD_CLEAR,IBPB,STIBP,L1DFL,SSBD>
>>  XSAVE Features=0x1<XSAVEOPT>
>>  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
>>  TSC: P-state invariant, performance statistics
>> real memory  = 274869518336 (262136 MB)
>> avail memory = 267244859392 (254864 MB)
>> Event timer "LAPIC" quality 600
>> ACPI APIC Table: <DELL   PE_SC3  >
>> FreeBSD/SMP: Multiprocessor System Detected: 32 CPUs
>> FreeBSD/SMP: 2 package(s) x 8 core(s) x 2 hardware threads
>> 
>> - Peter
> 
> I also have a Dell server, but older (R515, 128GB RAM, AMD 2x8 cores,
> ~140TB disk, ARC cap 64GB) that has experienced the same behaviour.  I
> get the impression the shutdown time is related to used swap, but I have
> not investigated further.  I think the system disk (with swap -
> mechanical!) activity light is on during this time.  Now running 11.3-R,
> but I'm sure I saw this with late 10.x as well.
> 
> Bengt