[Bug 293750] mv process hangs at 100% CPU (State R) in cache_fplookup / zap_lookup on ZFS
Date: Thu, 12 Mar 2026 00:26:25 UTC
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=293750
Bug ID: 293750
Summary: mv process hangs at 100% CPU (State R) in
cache_fplookup / zap_lookup on ZFS
Product: Base System
Version: 15.0-RELEASE
Hardware: amd64
OS: Any
Status: New
Severity: Affects Only Me
Priority: ---
Component: kern
Assignee: bugs@FreeBSD.org
Reporter: chris@ctgameinfo.com
Created attachment 268738
--> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=268738&action=edit
procstat -kka
A mv command involving a ZFS dataset has entered an uninterruptible state,
consuming 100% of a single CPU core. The process state is reported as R+ in ps
and shows a specific CPU (e.g., CPU8) in top, indicating it is spinning in
kernel space rather than sleeping on I/O.
The process does not respond to SIGKILL (-9).
After running these commands mv is unkillable and stuck at 100%
# cd /home/sites/x/
# mv www/www .
Additional info
------------
# uname -a
FreeBSD Akane 15.0-RELEASE-p4 FreeBSD 15.0-RELEASE-p4 GENERIC amd64
# zfs list
NAME USED AVAIL REFER MOUNTPOINT
<snip>
zroot/home/sites/x 836M 764G 554M /home/sites/x
zroot/home/sites/x/logs 6.35M 764G 6.35M /home/sites/x/logs
zroot/home/sites/x/www 276M 764G 276M /home/sites/x/www
<snip>
# ps -lp 84123
UID PID PPID C PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND
0 84123 76510 7 0 0 14072 2424 - R+ 1 23:36.51 mv www/www .
# truss -p 84123
<hangs with no output>
# ktrace -p 84123
#
# procstat -k 84123
PID TID COMM TDNAME KSTACK
84123 100542 mv - dsl_dir_tempreserve_space
dmu_tx_assign zfs_write zfs_freebsd_write VOP_WRITE_APV ktr_writerequest
ktr_submitrequest namei kern_renameat amd64_syscall fast_syscall_common
#top
last pid: 84552; load averages: 1.76, 1.77, 1.47
up 4+15:07:18 17:13:56
54 processes: 2 running, 52 sleeping
CPU: 3.3% user, 0.0% nice, 0.5% system, 0.0% interrupt, 96.2% idle
Mem: 877M Active, 5358M Inact, 24G Wired, 104K Buf, 909M Free
ARC: 18G Total, 9819M MFU, 7578M MRU, 74M Anon, 206M Header, 1035M Other
16G Compressed, 40G Uncompressed, 2.51:1 Ratio
Swap: 2048M Total, 2048M Free
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
84123 root 1 0 0 14M 2424K CPU9 9 25:11 99.02% mv
<snip>
# zpool status -v
pool: zroot
state: ONLINE
status: Some supported and requested features are not enabled on the pool.
The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(7) for details.
config:
NAME STATE READ WRITE CKSUM
zroot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
nda0p4 ONLINE 0 0 0
nda1p4 ONLINE 0 0 0
errors: No known data errors
# zfs get all zroot/home/sites/x/www | grep -v default
NAME PROPERTY VALUE
SOURCE
zroot/home/sites/x/www type filesystem
-
zroot/home/sites/x/www creation Fri Oct 3 15:19 2025
-
zroot/home/sites/x/www used 276M
-
zroot/home/sites/x/www available 763G
-
zroot/home/sites/x/www referenced 276M
-
zroot/home/sites/x/www compressratio 1.19x
-
zroot/home/sites/x/www mounted yes
-
zroot/home/sites/x/www mountpoint /home/sites/x/www inherited
from zroot/home
zroot/home/sites/x/www compression lz4
inherited from zroot
zroot/home/sites/x/www atime off
inherited from zroot
zroot/home/sites/x/www exec off
inherited from zroot/home/sites
zroot/home/sites/x/www setuid off
inherited from zroot/home/sites
zroot/home/sites/x/www createtxg 14688
-
zroot/home/sites/x/www version 5
-
zroot/home/sites/x/www utf8only off
-
zroot/home/sites/x/www normalization none
-
zroot/home/sites/x/www casesensitivity sensitive
-
zroot/home/sites/x/www guid 8959091402359551840
-
zroot/home/sites/x/www usedbysnapshots 0B
-
zroot/home/sites/x/www usedbydataset 276M
-
zroot/home/sites/x/www usedbychildren 0B
-
zroot/home/sites/x/www usedbyrefreservation 0B
-
zroot/home/sites/x/www objsetid 863
-
zroot/home/sites/x/www refcompressratio 1.19x
-
zroot/home/sites/x/www written 276M
-
zroot/home/sites/x/www logicalused 163M
-
zroot/home/sites/x/www logicalreferenced 163M
-
# zfs get all zroot/home/sites/x | grep -v default
NAME PROPERTY VALUE
SOURCE
zroot/home/sites/x type filesystem -
zroot/home/sites/x creation Thu Oct 2 22:28 2025 -
zroot/home/sites/x used 2.14G -
zroot/home/sites/x available 763G -
zroot/home/sites/x referenced 1.86G -
zroot/home/sites/x compressratio 6.00x -
zroot/home/sites/x mounted yes -
zroot/home/sites/x mountpoint /home/sites/x inherited from
zroot/home
zroot/home/sites/x compression lz4
inherited from zroot
zroot/home/sites/x atime off
inherited from zroot
zroot/home/sites/x exec off
inherited from zroot/home/sites
zroot/home/sites/x setuid off
inherited from zroot/home/sites
zroot/home/sites/x createtxg 2832 -
zroot/home/sites/x version 5 -
zroot/home/sites/x utf8only off -
zroot/home/sites/x normalization none -
zroot/home/sites/x casesensitivity sensitive -
zroot/home/sites/x guid 7802420260360558524 -
zroot/home/sites/x usedbysnapshots 0B -
zroot/home/sites/x usedbydataset 1.86G -
zroot/home/sites/x usedbychildren 282M -
zroot/home/sites/x usedbyrefreservation 0B -
zroot/home/sites/x objsetid 939 -
zroot/home/sites/x refcompressratio 6.35x -
zroot/home/sites/x written 1.86G -
zroot/home/sites/x logicalused 12.0G -
zroot/home/sites/x logicalreferenced 11.8G -
# sysctl kstat.zfs.misc.arcstats
kstat.zfs.misc.arcstats.abd_chunk_waste_size: 27941888
kstat.zfs.misc.arcstats.cached_only_in_progress: 0
kstat.zfs.misc.arcstats.arc_raw_size: 0
kstat.zfs.misc.arcstats.arc_sys_free: 0
kstat.zfs.misc.arcstats.arc_need_free: 0
kstat.zfs.misc.arcstats.demand_iohit_prescient_prefetch: 4
kstat.zfs.misc.arcstats.demand_hit_prescient_prefetch: 16
kstat.zfs.misc.arcstats.prescient_prefetch: 20
kstat.zfs.misc.arcstats.demand_iohit_predictive_prefetch: 4102
kstat.zfs.misc.arcstats.demand_hit_predictive_prefetch: 2861087
kstat.zfs.misc.arcstats.predictive_prefetch: 2906774
kstat.zfs.misc.arcstats.async_upgrade_sync: 1340
kstat.zfs.misc.arcstats.arc_dnode_limit: 3282970214
kstat.zfs.misc.arcstats.arc_meta_used: 2282039376
kstat.zfs.misc.arcstats.arc_prune: 0
kstat.zfs.misc.arcstats.arc_loaned_bytes: 0
kstat.zfs.misc.arcstats.arc_tempreserve: 0
kstat.zfs.misc.arcstats.arc_no_grow: 1
kstat.zfs.misc.arcstats.memory_available_bytes: 985264128
kstat.zfs.misc.arcstats.memory_free_bytes: 1691168768
kstat.zfs.misc.arcstats.memory_all_bytes: 33903443968
kstat.zfs.misc.arcstats.memory_indirect_count: 74
kstat.zfs.misc.arcstats.memory_direct_count: 0
kstat.zfs.misc.arcstats.memory_throttle_count: 0
kstat.zfs.misc.arcstats.l2_rebuild_log_blks: 0
kstat.zfs.misc.arcstats.l2_rebuild_bufs_precached: 0
kstat.zfs.misc.arcstats.l2_rebuild_bufs: 0
kstat.zfs.misc.arcstats.l2_rebuild_asize: 0
kstat.zfs.misc.arcstats.l2_rebuild_size: 0
kstat.zfs.misc.arcstats.l2_rebuild_lowmem: 0
kstat.zfs.misc.arcstats.l2_rebuild_cksum_lb_errors: 0
kstat.zfs.misc.arcstats.l2_rebuild_dh_errors: 0
kstat.zfs.misc.arcstats.l2_rebuild_io_errors: 0
kstat.zfs.misc.arcstats.l2_rebuild_unsupported: 0
kstat.zfs.misc.arcstats.l2_rebuild_success: 0
kstat.zfs.misc.arcstats.l2_data_to_meta_ratio: 0
kstat.zfs.misc.arcstats.l2_log_blk_count: 0
kstat.zfs.misc.arcstats.l2_log_blk_asize: 0
kstat.zfs.misc.arcstats.l2_log_blk_avg_asize: 0
kstat.zfs.misc.arcstats.l2_log_blk_writes: 0
kstat.zfs.misc.arcstats.l2_hdr_size: 0
kstat.zfs.misc.arcstats.l2_asize: 0
kstat.zfs.misc.arcstats.l2_size: 0
kstat.zfs.misc.arcstats.l2_io_error: 0
kstat.zfs.misc.arcstats.l2_cksum_bad: 0
kstat.zfs.misc.arcstats.l2_abort_lowmem: 0
kstat.zfs.misc.arcstats.l2_free_on_write: 0
kstat.zfs.misc.arcstats.l2_evict_l1cached: 0
kstat.zfs.misc.arcstats.l2_evict_reading: 0
kstat.zfs.misc.arcstats.l2_evict_lock_retry: 0
kstat.zfs.misc.arcstats.l2_writes_lock_retry: 0
kstat.zfs.misc.arcstats.l2_writes_error: 0
kstat.zfs.misc.arcstats.l2_writes_done: 0
kstat.zfs.misc.arcstats.l2_writes_sent: 0
kstat.zfs.misc.arcstats.l2_write_bytes: 0
kstat.zfs.misc.arcstats.l2_read_bytes: 0
kstat.zfs.misc.arcstats.l2_rw_clash: 0
kstat.zfs.misc.arcstats.l2_feeds: 0
kstat.zfs.misc.arcstats.l2_bufc_metadata_asize: 0
kstat.zfs.misc.arcstats.l2_bufc_data_asize: 0
kstat.zfs.misc.arcstats.l2_mfu_asize: 0
kstat.zfs.misc.arcstats.l2_mru_asize: 0
kstat.zfs.misc.arcstats.l2_prefetch_asize: 0
kstat.zfs.misc.arcstats.l2_misses: 0
kstat.zfs.misc.arcstats.l2_hits: 0
kstat.zfs.misc.arcstats.uncached_evictable_metadata: 0
kstat.zfs.misc.arcstats.uncached_evictable_data: 0
kstat.zfs.misc.arcstats.uncached_metadata: 0
kstat.zfs.misc.arcstats.uncached_data: 0
kstat.zfs.misc.arcstats.uncached_size: 0
kstat.zfs.misc.arcstats.mfu_ghost_evictable_metadata: 0
kstat.zfs.misc.arcstats.mfu_ghost_evictable_data: 4881907712
kstat.zfs.misc.arcstats.mfu_ghost_metadata: 0
kstat.zfs.misc.arcstats.mfu_ghost_data: 4881907712
kstat.zfs.misc.arcstats.mfu_ghost_size: 4881907712
kstat.zfs.misc.arcstats.mfu_evictable_metadata: 143363584
kstat.zfs.misc.arcstats.mfu_evictable_data: 9197573632
kstat.zfs.misc.arcstats.mfu_metadata: 361143808
kstat.zfs.misc.arcstats.mfu_data: 9354936832
kstat.zfs.misc.arcstats.mfu_size: 9716080640
kstat.zfs.misc.arcstats.mru_ghost_evictable_metadata: 0
kstat.zfs.misc.arcstats.mru_ghost_evictable_data: 5151820800
kstat.zfs.misc.arcstats.mru_ghost_metadata: 0
kstat.zfs.misc.arcstats.mru_ghost_data: 5151820800
kstat.zfs.misc.arcstats.mru_ghost_size: 5151820800
kstat.zfs.misc.arcstats.mru_evictable_metadata: 63363072
kstat.zfs.misc.arcstats.mru_evictable_data: 8079086592
kstat.zfs.misc.arcstats.mru_metadata: 567136768
kstat.zfs.misc.arcstats.mru_data: 8619606528
kstat.zfs.misc.arcstats.mru_size: 9186743296
kstat.zfs.misc.arcstats.anon_evictable_metadata: 0
kstat.zfs.misc.arcstats.anon_evictable_data: 0
kstat.zfs.misc.arcstats.anon_metadata: 9278976
kstat.zfs.misc.arcstats.anon_data: 119400960
kstat.zfs.misc.arcstats.anon_size: 128679936
kstat.zfs.misc.arcstats.other_size: 1100486960
kstat.zfs.misc.arcstats.bonus_size: 194260160
kstat.zfs.misc.arcstats.dnode_size: 709538280
kstat.zfs.misc.arcstats.dbuf_size: 196688520
kstat.zfs.misc.arcstats.metadata_size: 937559552
kstat.zfs.misc.arcstats.data_size: 18093944320
kstat.zfs.misc.arcstats.hdr_size: 243992864
kstat.zfs.misc.arcstats.overhead_size: 1214676992
kstat.zfs.misc.arcstats.uncompressed_size: 53352812544
kstat.zfs.misc.arcstats.compressed_size: 17816826880
kstat.zfs.misc.arcstats.size: 20403925584
kstat.zfs.misc.arcstats.c_max: 32829702144
kstat.zfs.misc.arcstats.c_min: 1059482624
kstat.zfs.misc.arcstats.c: 20487444490
kstat.zfs.misc.arcstats.pm: 2147483648
kstat.zfs.misc.arcstats.pd: 2083816569
kstat.zfs.misc.arcstats.meta: 970340663
kstat.zfs.misc.arcstats.hash_chain_max: 6
kstat.zfs.misc.arcstats.hash_chains: 104985
kstat.zfs.misc.arcstats.hash_collisions: 9094239
kstat.zfs.misc.arcstats.hash_elements_max: 1017607
kstat.zfs.misc.arcstats.hash_elements: 1017607
kstat.zfs.misc.arcstats.evict_l2_skip: 0
kstat.zfs.misc.arcstats.evict_l2_ineligible: 99614720
kstat.zfs.misc.arcstats.evict_l2_eligible_mru: 10961854464
kstat.zfs.misc.arcstats.evict_l2_eligible_mfu: 162220245504
kstat.zfs.misc.arcstats.evict_l2_eligible: 173182099968
kstat.zfs.misc.arcstats.evict_l2_cached: 0
kstat.zfs.misc.arcstats.evict_not_enough: 0
kstat.zfs.misc.arcstats.evict_skip: 88612
kstat.zfs.misc.arcstats.access_skip: 23
kstat.zfs.misc.arcstats.mutex_miss: 93
kstat.zfs.misc.arcstats.deleted: 1282149
kstat.zfs.misc.arcstats.uncached_hits: 380244
kstat.zfs.misc.arcstats.mfu_ghost_hits: 65466
kstat.zfs.misc.arcstats.mfu_hits: 18867017351
kstat.zfs.misc.arcstats.mru_ghost_hits: 92342
kstat.zfs.misc.arcstats.mru_hits: 46794883
kstat.zfs.misc.arcstats.prefetch_metadata_misses: 25067
kstat.zfs.misc.arcstats.prefetch_metadata_iohits: 26586
kstat.zfs.misc.arcstats.prefetch_metadata_hits: 187438
kstat.zfs.misc.arcstats.prefetch_data_misses: 1160285
kstat.zfs.misc.arcstats.prefetch_data_iohits: 0
kstat.zfs.misc.arcstats.prefetch_data_hits: 1507418
kstat.zfs.misc.arcstats.demand_metadata_misses: 42409
kstat.zfs.misc.arcstats.demand_metadata_iohits: 3145
kstat.zfs.misc.arcstats.demand_metadata_hits: 12904666598
kstat.zfs.misc.arcstats.demand_data_misses: 531109
kstat.zfs.misc.arcstats.demand_data_iohits: 1178
kstat.zfs.misc.arcstats.demand_data_hits: 6007831233
kstat.zfs.misc.arcstats.misses: 1758870
kstat.zfs.misc.arcstats.iohits: 30909
kstat.zfs.misc.arcstats.hits: 18914192788
# vmstat -z
ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP XDOM
kstack_cache: 16384, 0, 673, 22, 8227, 0, 0, 0
buffer arena-40: 4096, 0, 0, 0, 0, 0, 0, 0
buffer arena-81: 8192, 0, 0, 0, 0, 0, 0, 0
buffer arena-12: 12288, 0, 0, 0, 0, 0, 0, 0
buffer arena-16: 16384, 0, 10, 0, 10, 0, 0, 0
buffer arena-20: 20480, 0, 0, 0, 0, 0, 0, 0
buffer arena-24: 24576, 0, 0, 0, 0, 0, 0, 0
buffer arena-28: 28672, 0, 0, 0, 0, 0, 0, 0
buffer arena-32: 32768, 0, 4, 0, 4, 0, 0, 0
buffer arena-36: 36864, 0, 0, 0, 0, 0, 0, 0
buffer arena-40: 40960, 0, 0, 0, 0, 0, 0, 0
buffer arena-45: 45056, 0, 0, 0, 0, 0, 0, 0
buffer arena-49: 49152, 0, 0, 0, 0, 0, 0, 0
buffer arena-53: 53248, 0, 0, 0, 0, 0, 0, 0
buffer arena-57: 57344, 0, 0, 0, 0, 0, 0, 0
buffer arena-61: 61440, 0, 0, 0, 0, 0, 0, 0
buffer arena-65: 65536, 0, 0, 0, 0, 0, 0, 0
buf free cache: 432, 0, 14, 196, 14, 0, 0, 0
vm pgcache: 4096, 0, 844303, 3296,1004668630,3061, 0, 0
vm pgcache: 4096, 0, 5644515, 718,32595219, 610, 0, 0
UMA Kegs: 384, 0, 157, 6, 157, 0, 0, 0
UMA Zones: 2048, 0, 183, 7, 183, 0, 0, 0
UMA Slabs 0: 80, 0, 4973917, 372927,19957826, 0, 0, 0
UMA Slabs 1: 112, 0, 11, 24, 11, 0, 0, 0
UMA Hash: 256, 0, 0, 0, 0, 0, 0, 0
2 Bucket: 32, 0, 935, 8389, 3046611, 0, 0, 0
4 Bucket: 48, 0, 36, 4668, 343338, 0, 0, 0
8 Bucket: 80, 0, 4346, 2004, 3539030, 0, 0, 0
16 Bucket: 144, 0, 1336, 2696, 2918946, 0, 0, 0
32 Bucket: 256, 0, 7401, 504, 7134107, 1, 0, 0
64 Bucket: 512, 0, 6764, 324,62634999, 185, 0, 0
128 Bucket: 1024, 0, 1600, 175,11047021, 211, 0, 0
256 Bucket: 2048, 0, 8416, 4900,29106739,2102558, 0,
0
SMR SHARED: 24, 0, 8, 759, 8, 0, 0, 0
SMR CPU: 32, 0, 8, 759, 8, 0, 0, 0
vmem: 1856, 0, 2, 7, 2, 0, 0, 0
vmem btag: 56, 0, 132182, 42334, 1138834,1214, 0, 0
VM OBJECT: 248, 0, 238509, 93395,51578461, 0, 0, 0
RADIX NODE: 152, 0, 154419, 21208,82398204, 0, 0, 0
KMAP ENTRY: 96, 0, 33, 6, 35, 0, 0, 0
MAP ENTRY: 96, 0, 4709, 7387,17333200, 0, 0, 0
VMSPACE: 584, 0, 55, 267, 88176, 0, 0, 0
fakepg: 88, 0, 0, 138, 4, 0, 0, 0
pcpu-4: 4, 0, 0, 0, 0, 0, 0, 0
pcpu-8: 8, 0, 4144, 2000, 4146, 0, 0, 0
pcpu-16: 16, 0, 44, 2772, 44, 0, 0, 0
pcpu-32: 32, 0, 0, 0, 0, 0, 0, 0
pcpu-64: 64, 0, 467, 813, 467, 0, 0, 0
malloc-16: 16, 0, 19064, 4120,8363168925, 0, 0, 0
malloc-32: 32, 0, 271616, 96808,4697673642, 0, 0, 0
malloc-64: 64, 0, 281350, 181889,14155325777, 0, 0,
0
malloc-128: 128, 0, 1050510, 116454,14491626167, 0, 0,
0
malloc-256: 256, 0, 125059, 91541,1197604864, 0, 0, 0
malloc-384: 384, 0, 565691, 106349,310312929, 0, 0, 0
malloc-512: 512, 0, 240, 1432,156315676, 0, 0, 0
malloc-1024: 1024, 0, 2005, 68367,119633993, 0, 0, 0
malloc-2048: 2048, 0, 677, 213,102092015, 0, 0, 0
malloc-4096: 4096, 0, 33684, 69,262040425, 0, 0, 0
malloc-8192: 8192, 0, 86, 33, 8038, 0, 0, 0
malloc-16384: 16384, 0, 29, 30,56428258, 0, 0, 0
malloc-32768: 32768, 0, 30, 11, 812, 0, 0, 0
malloc-65536: 65536, 0, 14, 13, 81454, 0, 0, 0
SLEEPQUEUE: 80, 0, 809, 1111, 809, 0, 0, 0
kenv: 258, 0, 18, 342, 10063, 0, 0, 0
Files: 80, 0, 866, 2434,2179224078, 0, 0, 0
filedesc0: 1072, 0, 77, 161, 95038, 0, 0, 0
PWD: 40, 0, 24, 4319, 2967629, 0, 0, 0
rl_entry: 40, 0, 20428, 4721,2127216242, 0, 0, 0
rangeset pctrie nodes: 152, 0, 0, 0, 0, 0, 0, 0
TURNSTILE: 136, 0, 809, 388, 809, 0, 0, 0
lkpicurr: 168, 0, 2, 3142, 3, 0, 0, 0
lkpimm: 56, 0, 1, 3167, 2, 0, 0, 0
umtx pi: 96, 0, 0, 0, 0, 0, 0, 0
umtx_shm: 88, 0, 0, 0, 0, 0, 0, 0
swrbuf: 2360, 192, 0, 0, 0, 0, 0, 0
swwbuf: 2360, 96, 0, 0, 0, 0, 0, 0
nfspbuf: 2360, 192, 0, 0, 0, 0, 0, 0
vnpbuf: 2360, 768, 0, 0, 0, 0, 0, 0
clpbuf: 2360, 192, 0, 0, 0, 0, 0, 0
pbuf: 2360, 16, 0, 1170, 0, 0, 0, 0
FPU_save_area: 2440, 0, 675, 75, 444110, 0, 0, 0
MAC labels: 40, 0, 0, 0, 0, 0, 0, 0
PROC: 1368, 0, 76, 133, 95037, 0, 0, 0
PGRP: 120, 0, 37, 1481, 12981, 0, 0, 0
THREAD: 1860, 0, 673, 135, 8227, 0, 0, 0
cpuset: 200, 0, 39, 713, 39, 0, 0, 0
domainset: 40, 0, 0, 0, 0, 0, 0, 0
pkru ranges: 24, 0, 0, 0, 0, 0, 0, 0
audit_record: 1280, 0, 0, 0, 0, 0, 0, 0
mbuf_packet: 256, 0, 20, 4298,1221253241, 0, 0, 0
mbuf: 256,12939731, 3211, 9886,3808837764, 0, 0,
0
mbuf_cluster: 2048,2021833, 7389, 3917, 6878252, 0, 0, 0
mbuf_jumbo_page: 4096,1010916, 46, 4549,1083988772, 0, 0, 0
mbuf_jumbo_9k: 9216, 299530, 0, 0, 0, 0, 0, 0
mbuf_jumbo_16k: 16384, 168486, 0, 0, 0, 0, 0, 0
epoch_record pcpu: 256, 0, 4, 60, 4, 0, 0, 0
ttyinq: 160, 0, 420, 530, 795, 0, 0, 0
ttyoutq: 256, 0, 219, 531, 417, 0, 0, 0
skbuff: 1808, 0, 0, 0, 0, 0, 0, 0
IOMMU_MAP_ENTRY: 112, 0, 0, 0, 0, 0, 0, 0
DEVCTL: 1024, 1000, 0, 100, 517, 0, 0, 0
g_bio: 376, 0, 0, 1080,106457875, 0, 0, 0
linux_dma_pctrie: 152, 0, 0, 0, 0, 0, 0, 0
linux_dma_object: 32, 0, 0, 0, 0, 0, 0, 0
vtnet_tx_hdr: 24, 0, 0, 168, 0, 0, 0, 0
cryptop: 280, 0, 0, 0, 0, 0, 0, 0
taskq_zone: 160, 0, 0, 2184,13319951, 0, 0, 0
VNODE: 440, 0, 540562, 90824,3649099928, 0, 0, 0
BUF TRIE: 152, 0, 5, 105945, 5, 0, 0, 0
S VFS Cache: 104, 0, 544663, 184559,964582813, 0, 0, 0
STS VFS Cache: 144, 0, 0, 0, 0, 0, 0, 0
L VFS Cache: 320, 0, 58432, 620, 61650, 0, 0, 0
LTS VFS Cache: 360, 0, 0, 0, 0, 0, 0, 0
NAMEI: 1024, 0, 2, 234,9577136004, 0, 0, 0
NCLNODE: 608, 0, 0, 0, 0, 0, 0, 0
TMPFS node: 240, 0, 0, 0, 0, 0, 0, 0
DIRHASH: 1024, 0, 0, 0, 0, 0, 0, 0
pipe: 736, 0, 6, 247, 50894, 0, 0, 0
AIO: 208, 0, 3, 510, 15, 0, 0, 0
AIOCB: 552, 0, 0, 385,11454075, 0, 0, 0
AIOLIO: 272, 0, 0, 0, 0, 0, 0, 0
Mountpoints: 2816, 0, 24, 20, 24, 0, 0, 0
zfs_btree_leaf_cache: 4096, 0, 420, 68222,100219137, 0, 0, 0
metaslab_alloc_trace_cache:72, 0, 0, 0, 0, 0, 0, 0
brt_entry_cache: 168, 0, 0, 2112, 9566, 0, 0, 0
ddt_cache: 29240, 0, 5, 1, 5, 0, 0, 0
ddt_entry_flat_cache: 184, 0, 0, 0, 0, 0, 0, 0
ddt_entry_trad_cache: 368, 0, 0, 0, 0, 0, 0, 0
ddt_log_entry_flat_cache:144, 0, 0, 0, 0, 0, 0, 0
ddt_log_entry_trad_cache:328, 0, 0, 0, 0, 0, 0, 0
zio_cache: 1152, 0, 16, 78300,8481397489, 0, 0, 0
zio_link_cache: 48, 0, 0, 78792,767079482, 0, 0, 0
zio_buf_comb_512: 512, 0, 1366012, 1578060,1420577179, 0, 0, 0
zio_buf_comb_1024: 1024, 0, 9512, 6544, 9438728, 0, 0, 0
zio_buf_comb_1536: 1536, 0, 6475, 5150, 5112094, 0, 0, 0
zio_buf_comb_2048: 2048, 0, 4294, 3254, 3461483, 0, 0, 0
zio_buf_comb_3072: 3072, 0, 14741, 25079,26768368, 0, 0, 0
zio_buf_comb_4096: 4096, 0, 72352, 11097,36431683, 0, 0, 0
zio_buf_comb_6144: 6144, 0, 9, 721, 6483480, 0, 0, 0
zio_buf_comb_8192: 8192, 0, 4, 379,15771599, 0, 0, 0
zio_buf_comb_12288: 12288, 0, 4, 81, 4786468, 0, 0, 0
zio_buf_comb_16384: 16384, 0, 34230, 2904,32898875, 0, 0, 0
zio_buf_comb_24576: 24576, 0, 4, 58, 1654310, 0, 0, 0
zio_buf_comb_32768: 32768, 0, 6, 31, 928036, 0, 0, 0
zio_buf_comb_49152: 49152, 0, 19, 34, 1788802, 0, 0, 0
zio_buf_comb_65536: 65536, 0, 30, 44, 891995, 0, 0, 0
zio_buf_comb_98304: 98304, 0, 129, 48, 1447082, 0, 0, 0
zio_buf_comb_131072: 131072, 0, 5177, 2171,15216190, 0, 0, 0
zio_buf_comb_196608: 196608, 0, 0, 0, 0, 0, 0, 0
zio_buf_comb_262144: 262144, 0, 0, 1, 17, 0, 0, 0
zio_buf_comb_393216: 393216, 0, 0, 0, 0, 0, 0, 0
zio_buf_comb_524288: 524288, 0, 0, 0, 0, 0, 0, 0
zio_buf_comb_786432: 786432, 0, 0, 0, 0, 0, 0, 0
zio_buf_comb_1048576:1048576, 0, 0, 0, 0, 0, 0, 0
zio_buf_comb_1572864:1572864, 0, 0, 0, 0, 0, 0, 0
zio_buf_comb_2097152:2097152, 0, 0, 0, 0, 0, 0, 0
zio_buf_comb_3145728:3145728, 0, 0, 0, 0, 0, 0, 0
zio_buf_comb_4194304:4194304, 0, 0, 0, 0, 0, 0, 0
zio_buf_comb_6291456:6291456, 0, 0, 0, 0, 0, 0, 0
zio_buf_comb_8388608:8388608, 0, 0, 0, 0, 0, 0, 0
zio_buf_comb_12582912:12582912, 0, 0, 0, 0, 0, 0,
0
zio_buf_comb_16777216:16777216, 0, 0, 0, 0, 0, 0,
0
abd_chunk: 4096, 0, 4318036, 632,53939212, 0, 0, 0
sa_cache: 264, 0, 540406, 82604,49065995, 0, 0, 0
dnode_t: 760, 0, 932459, 55136,12137113, 0, 0, 0
arc_buf_hdr_t_full: 232, 0, 997020, 50180,421922938, 0, 0, 0
arc_buf_hdr_t_l2only: 96, 0, 0, 0, 0, 0, 0, 0
arc_buf_t: 32, 0, 40429, 81665,446777460, 0, 0, 0
dmu_buf_impl_t: 280, 0, 581171, 165449,486635758, 0, 0, 0
dbuf_dirty_record_t: 376, 0, 1643, 96397,158527418, 0, 0, 0
zil_lwb_cache: 376, 0, 58, 1182, 4083369, 0, 0, 0
zil_zcw_cache: 80, 0, 0, 1900, 3735150, 0, 0, 0
sio_cache_0: 136, 0, 0, 0, 0, 0, 0, 0
sio_cache_1: 152, 0, 0, 0, 0, 0, 0, 0
sio_cache_2: 168, 0, 0, 0, 0, 0, 0, 0
zap_name: 328, 0, 0, 408,4147714242, 0, 0, 0
zap_attr_cache: 288, 0, 0, 364, 27382, 0, 0, 0
zap_name_long: 1096, 0, 0, 0, 0, 0, 0, 0
zap_attr_long_cache: 1056, 0, 0, 133,260549647, 0, 0, 0
zfs_znode_cache: 456, 0, 540406, 96737,49065995, 0, 0, 0
ksiginfo: 112, 0, 203, 2533, 420371, 0, 0, 0
itimer: 352, 0, 0, 0, 0, 0, 0, 0
KNOTE: 160, 0, 160, 1340,67699396, 0, 0, 0
socket: 1024,1034651, 275, 3769,10474914, 0, 0, 0
IPsec SA lft_c: 16, 0, 0, 0, 0, 0, 0, 0
unpcb: 320,1034651, 45, 3795, 8255784, 0, 0, 0
ripcb: 376, 0, 1, 369, 345, 0, 0, 0
tcp_inpcb: 1304, 0, 208, 203, 929640, 0, 0, 0
udp_inpcb: 408, 0, 18, 531, 1289098, 0, 0, 0
udplite_inpcb: 408, 0, 0, 0, 0, 0, 0, 0
routing nhops: 256, 0, 8, 187, 13, 0, 0, 0
tcpreass: 48, 126364, 0, 1596, 316394, 0, 0, 0
tcp_log: 408,5000000, 0, 0, 0, 0, 0, 0
tcp_log_id_bucket: 176, 0, 0, 0, 0, 0, 0, 0
tcp_log_id_node: 120, 0, 0, 0, 0, 0, 0, 0
rtentry: 168, 0, 11, 181, 13, 0, 0, 0
ipq: 56, 51200, 0, 576, 7, 0, 0, 0
syncache: 176, 15360, 1, 4392, 1182096, 0, 0, 0
hostcache: 64, 15360, 3085, 11972, 287572,6101, 0, 0
sackhole: 32, 0, 0, 2394, 1017279, 0, 0, 0
tfo: 4, 0, 0, 0, 0, 0, 0, 0
tfo_ccache_entries: 80, 0, 0, 0, 0, 0, 0, 0
ada_ccb: 272, 0, 0, 0, 0, 0, 0, 0
da_ccb: 536, 0, 0, 0, 0, 0, 0, 0
swpctrie: 152,4043676, 0, 0, 0, 0, 0, 0
swblk: 136,4043673, 0, 0, 0, 0, 0, 0
IPFW counters: 16, 0, 60, 2756, 71, 0, 0, 0
IPFW dynamic states data:88, 16384, 10, 2980, 630566, 0, 0, 0
IPFW parent dynamic states:32, 4096, 0, 0, 0, 0, 0, 0
IPFW IPv4 dynamic states:48, 0, 10, 3350, 630566, 0, 0, 0
IPFW IPv6 dynamic states:72, 0, 0, 0, 0, 0, 0, 0
# iostat -x 1
extended device statistics
device r/s w/s kr/s kw/s ms/r ms/w ms/o ms/t qlen %b
nda0 2 43 38.5 1362.2 0 0 0 0 0 0
nda1 2 43 38.7 1362.2 0 0 0 0 0 0
pass0 0 0 0.0 0.0 0 0 0 0 0 0
pass1 0 0 0.0 0.0 0 0 0 0 0 0
extended device statistics
device r/s w/s kr/s kw/s ms/r ms/w ms/o ms/t qlen %b
nda0 0 4 0.0 35.5 0 0 0 0 0 0
nda1 0 4 0.0 35.5 0 0 0 0 0 0
pass0 0 0 0.0 0.0 0 0 0 0 0 0
pass1 0 0 0.0 0.0 0 0 0 0 0 0
extended device statistics
device r/s w/s kr/s kw/s ms/r ms/w ms/o ms/t qlen %b
nda0 0 388 0.0 33440.7 0 0 0 0 0 8
nda1 0 376 0.0 33440.7 0 0 0 0 0 9
pass0 0 0 0.0 0.0 0 0 0 0 0 0
pass1 0 0 0.0 0.0 0 0 0 0 0 0
extended device statistics
device r/s w/s kr/s kw/s ms/r ms/w ms/o ms/t qlen %b
nda0 0 0 0.0 0.0 0 0 0 0 0 0
nda1 0 0 0.0 0.0 0 0 0 0 0 0
pass0 0 0 0.0 0.0 0 0 0 0 0 0
pass1 0 0 0.0 0.0 0 0 0 0 0 0
extended device statistics
device r/s w/s kr/s kw/s ms/r ms/w ms/o ms/t qlen %b
nda0 0 30 0.0 308.0 0 0 0 0 0 0
nda1 0 30 0.0 308.0 0 0 0 0 0 0
pass0 0 0 0.0 0.0 0 0 0 0 0 0
pass1 0 0 0.0 0.0 0 0 0 0 0 0
^C extended device statistics
device r/s w/s kr/s kw/s ms/r ms/w ms/o ms/t qlen %b
nda0 0 7 0.0 64.8 0 0 0 0 0 0
nda1 0 7 0.0 64.8 0 0 0 0 0 0
pass0 0 0 0.0 0.0 0 0 0 0 0 0
pass1 0 0 0.0 0.0 0 0 0 0 0 0
# dtrace -n 'fbt::zap_lookup:entry /execname == "mv"/ { self->ts = timestamp; }
fbt::zap_lookup:return /self->ts/ { printf("lat: %d", timestamp - self->ts); }'
| more
dtrace: description 'fbt::zap_lookup:entry ' matched 2 probes
CPU ID FUNCTION:NAME
5 76354 zap_lookup:return lat: 1342
5 76354 zap_lookup:return lat: 601
5 76354 zap_lookup:return lat: 311
5 76354 zap_lookup:return lat: 330
5 76354 zap_lookup:return lat: 471
5 76354 zap_lookup:return lat: 320
5 76354 zap_lookup:return lat: 321
5 76354 zap_lookup:return lat: 321
5 76354 zap_lookup:return lat: 421
5 76354 zap_lookup:return lat: 350
5 76354 zap_lookup:return lat: 291
5 76354 zap_lookup:return lat: 330
5 76354 zap_lookup:return lat: 411
5 76354 zap_lookup:return lat: 320
5 76354 zap_lookup:return lat: 310
5 76354 zap_lookup:return lat: 340
5 76354 zap_lookup:return lat: 410
5 76354 zap_lookup:return lat: 341
5 76354 zap_lookup:return lat: 290
5 76354 zap_lookup:return lat: 351
5 76354 zap_lookup:return lat: 421
5 76354 zap_lookup:return lat: 351
5 76354 zap_lookup:return lat: 311
5 76354 zap_lookup:return lat: 371
5 76354 zap_lookup:return lat: 431
5 76354 zap_lookup:return lat: 321
5 76354 zap_lookup:return lat: 291
5 76354 zap_lookup:return lat: 321
5 76354 zap_lookup:return lat: 441
5 76354 zap_lookup:return lat: 381
5 76354 zap_lookup:return lat: 301
5 76354 zap_lookup:return lat: 331
5 76354 zap_lookup:return lat: 411
5 76354 zap_lookup:return lat: 330
5 76354 zap_lookup:return lat: 301
5 76354 zap_lookup:return lat: 692
5 76354 zap_lookup:return lat: 410
5 76354 zap_lookup:return lat: 321
5 76354 zap_lookup:return lat: 291
5 76354 zap_lookup:return lat: 320
5 76354 zap_lookup:return lat: 411
5 76354 zap_lookup:return lat: 350
5 76354 zap_lookup:return lat: 301
5 76354 zap_lookup:return lat: 320
dtrace: failed to write to <stdout>: Broken pipe
Sorry about the long report. Trying to provide anything that could be useful
before I have to reboot.
--
You are receiving this mail because:
You are the assignee for the bug.