[Bug 242427] pmap_remove() sometimes is very slow causing 10+ minutes long reboots

[bugzilla-noreply at freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=242427

--- Comment #6 from Peter Eriksson <pen at lysator.liu.se> ---
(In reply to Konstantin Belousov from comment #5)

Yes, you are correct (and I was wrong). Sorry about that. 

I've rewritten the timing tests now to use nanouptime() instead, and now things
look slightly different. Albeit still takes a long time...

Here's a sample from a server that has been up for some hours:

       keg_free_slab: keg->uk_freef(mem) {page_free} took 2077 µs
       keg_free_slab: keg->uk_freef(mem) {page_free} took 2123 µs
       keg_free_slab: keg->uk_freef(mem) {page_free} took 2107 µs
       keg_free_slab: keg->uk_freef(mem) {page_free} took 2190 µs
      keg_drain: while()-keg_free_slab-loop took 163 s (163765748 µs) [240297
loops, 4 slow, avg=681 µs, min=246 µs, max=7922 µs]
     zone_drain_wait(): zone_foreach_keg(zone, &keg_drain) took 163 s
    zone_dtor(): zone_drain_wait(zone, M_WAITOK) took 163 s
   zone_free_item(zone=UMA Zones): zone->uz_dtor() took 163 s
  uma_zdestroy(zio_buf_16384) took 163 s
 kmem_cache_destroy: uma_zdestroy(0xfffff80346601880) [zio_buf_16384] took 164
seconds
zio_fini: kmem_cache_destroy(zio_buf_cache[28]) took 164 seconds

keg_free_slab only prints the keg->uk_freef(mem) (calls page_free which calls
kmem_free) call timings if it takes over 2000µs. Most of them are not shown,
the majority seems to be around 1000-1500µs.


Anyway, I wonder if one couldn't just skip all this freeing of kernel memory at
reboot/shutdown time. Perhaps it could conditionally be disabled via a sysctl
setting? 

In order to test this I added such a sysctl and with the calls to
kmem_cache_destroy() in zio_fini() disabled this machine reboots in under 1
minute instead of atleast 10 minutes (for a newly booted server). (The LSI SAS
HBA shutdown and the ZFS unmounting takes the most of that time).


(I suppose the current code is needed for the case where you have dynamically
loaded zfs and wishes to kldunload it (without rebooting) - or one will have a
massive memory leak, but since we're rebooting almost immediately after this I
don't really see why we have to spend 10-60 minutes freeing all this memory
:-).

zio.fini() is in /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c
The rest are in /usr/src/sys/vm/uma_core.c & vm_kern.c


A slightly longer output (verbose-level 2 so skipping some timing measurements
in keg_free_slab and below that might inflate the timing numbers) for reference
from a freshly rebooted server:

Freeing ZFS ZIO caches:

keg_drain: while()-keg_free_slab-loop took 994 ms (993875 µs) [1450 loops, 0
slow, avg=685 µs, min=383 µs, max=1713 µs, 0 zero]
zone_drain_wait(): zone_foreach_keg(zone, &keg_drain) took 7319 µs
zone_dtor(): zone_drain_wait(zone, M_WAITOK) took 14 ms
uma_zdestroy(zio_buf_8192) took 19 ms

keg_drain: while()-keg_free_slab-loop took 15 s (15886587 µs) [23217 loops, 0
slow, avg=684 µs, min=279 µs, max=1745 µs, 0 zero]
zone_drain_wait(): zone_foreach_keg(zone, &keg_drain) took 15 s
zone_dtor(): zone_drain_wait(zone, M_WAITOK) took 15 s
uma_zdestroy(zio_buf_12288) took 15 s
kmem_cache_destroy: uma_zdestroy(0xfffff803465f6980) [zio_buf_12288] took 16
seconds
zio_fini: kmem_cache_destroy(zio_buf_cache[20]) took 16 seconds

keg_drain: while()-keg_free_slab-loop took 61 s (61698566 µs) [86029 loops, 11
slow, avg=717 µs, min=265 µs, max=2252 µs, 0 zero]
zone_drain_wait(): zone_foreach_keg(zone, &keg_drain) took 61 s
zone_dtor(): zone_drain_wait(zone, M_WAITOK) took 61 s
uma_zdestroy(zio_buf_16384) took 61 s
kmem_cache_destroy: uma_zdestroy(0xfffff803465f6880) [zio_buf_16384] took 62
seconds
zio_fini: kmem_cache_destroy(zio_buf_cache[28]) took 62 seconds

keg_drain: while()-keg_free_slab-loop took 87 s (86986351 µs) [123793 loops, 24
slow, avg=702 µs, min=297 µs, max=2398 µs, 0 zero]
zone_drain_wait(): zone_foreach_keg(zone, &keg_drain) took 87 s
zone_dtor(): zone_drain_wait(zone, M_WAITOK) took 87 s
uma_zdestroy(zio_buf_131072) took 87 s
kmem_cache_destroy: uma_zdestroy(0xfffff8034c904640) [zio_buf_131072] took 87
seconds
zio_fini: kmem_cache_destroy(zio_buf_cache[224]) took 87 seconds

keg_drain: while()-keg_free_slab-loop took 344 ms (5340963 µs) [7730 loops, 0
slow, avg=690 µs, min=371 µs, max=1739 µs, 0 zero]
zone_drain_wait(): zone_foreach_keg(zone, &keg_drain) took 357 ms
zone_dtor(): zone_drain_wait(zone, M_WAITOK) took 364 ms
uma_zdestroy(zio_data_buf_131072) took 370 ms
kmem_cache_destroy: uma_zdestroy(0xfffff8034c904600) [zio_data_buf_131072] took
6 seconds
zio_fini: kmem_cache_destroy(zio_data_buf_cache[224]) took 6 seconds

-- 
You are receiving this mail because:
You are the assignee for the bug.