ZFS directory not accessable
Kai Kockro
k.kockro at myphotobook.de
Wed Feb 19 11:39:46 UTC 2014
zdb -ddddddd umbrella/usr/home 1027
Dataset umbrella/usr/home [ZPL], ID 52, cr_txg 48, 100G, 114028 objects,
rootbp DVA[0]=<0:6c023c400:200> DVA[1]=<0:c003b8a00:200> [L0 DMU objset]
fletcher4 lzjb LE contiguous unique double size=800L/200P
birth=2947973L/2947973P fill=114028
cksum=1d607c8fa9:939a0909e55:192c77959d738:310b5a95037557
Object lvl iblk dblk dsize lsize %full type
1027 1 16K 3.00K 2K 3.00K 100.00 ZFS directory
(K=inherit) (Z=inherit)
168 bonus System attributes
dnode flags: USED_BYTES USERUSED_ACCOUNTED
dnode maxblkid: 0
path /user/.SkypeOLD/user_name
uid 1001
gid 0
atime Tue Feb 11 12:45:46 2014
mtime Fri Feb 14 07:36:26 2014
ctime Fri Feb 14 13:43:50 2014
crtime Wed Feb 29 21:32:14 2012
gen 15787
mode 40755
size 41
parent 978
links 5
pflags 40800000144
Assertion failed: (uintptr_t)&((uint64_t
*)(zap)->zap_u.zap_fat.zap_phys)
[(1<<(((zap)->zap_u.zap_fat.zap_block_shift) - 3 - 1)) +
(1<<(((zap)->zap_u.zap_fat.zap_block_shift) - 3 - 1))] -
(uintptr_t)zap->zap_u.zap_fat.zap_phys == zap->zap_dbuf->db_size (0x800
== 0xc00), file
/usr/src/cddl/lib/libzpool/../../../sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zap_micro.c,
line 450.
Abbruch(core dumped)
I did a bt on this core dump:
(gdb) bt
#0 0x0000000801c8f45c in thr_kill () from /lib/libc.so.7
#1 0x0000000801d338db in abort () from /lib/libc.so.7
#2 0x0000000801916ede in zap_lockdir () from /lib/libzpool.so.2
#3 0x0000000801917214 in zap_get_stats () from /lib/libzpool.so.2
#4 0x00000000004072e0 in ?? ()
#5 0x000000000040758a in ?? ()
#6 0x0000000000409dd6 in ?? ()
#7 0x000000000040a424 in ?? ()
#8 0x000000000040c0f8 in ?? ()
#9 0x0000000000404f11 in ?? ()
#10 0x000000080062d000 in ?? ()
#11 0x0000000000000000 in ?? ()
kgdb
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...
Reading symbols from /boot/kernel/zfs.ko...Reading symbols from
/boot/kernel/zfs.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/zfs.ko
Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from
/boot/kernel/opensolaris.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/opensolaris.ko
Reading symbols from /boot/modules/nvidia.ko...done.
Loaded symbols for /boot/modules/nvidia.ko
Reading symbols from /boot/modules/vboxdrv.ko...done.
Loaded symbols for /boot/modules/vboxdrv.ko
Reading symbols from /boot/kernel/umodem.ko...Reading symbols from
/boot/kernel/umodem.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/umodem.ko
Reading symbols from /boot/modules/vboxnetflt.ko...done.
Loaded symbols for /boot/modules/vboxnetflt.ko
Reading symbols from /boot/kernel/netgraph.ko...Reading symbols from
/boot/kernel/netgraph.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/netgraph.ko
Reading symbols from /boot/kernel/ng_ether.ko...Reading symbols from
/boot/kernel/ng_ether.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ng_ether.ko
Reading symbols from /boot/modules/vboxnetadp.ko...done.
Loaded symbols for /boot/modules/vboxnetadp.ko
Reading symbols from /boot/kernel/linux.ko...Reading symbols from
/boot/kernel/linux.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/linux.ko
#0 fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:603
603 movq %r12,%rdi /* function */
(kgdb) tid 100558
[Switching to thread 426 (Thread 100558)]#0 sched_switch
(td=0xfffffe000c00c920, newtd=0xfffffe0007212920, flags=<value optimized
out>) at /usr/src/sys/kern/sched_ule.c:1904
1904 cpuid = PCPU_GET(cpuid);
(kgdb) bt
#0 sched_switch (td=0xfffffe000c00c920, newtd=0xfffffe0007212920,
flags=<value optimized out>) at /usr/src/sys/kern/sched_ule.c:1904
#1 0xffffffff808d6ee4 in mi_switch (flags=260, newtd=0x0) at
/usr/src/sys/kern/kern_synch.c:485
#2 0xffffffff809130a2 in sleepq_wait (wchan=0xfffffe00c0af0e00, pri=0)
at /usr/src/sys/kern/subr_sleepqueue.c:621
#3 0xffffffff808d5d80 in _sx_slock_hard (sx=0xfffffe00c0af0e00,
opts=<value optimized out>, file=<value optimized out>, line=<value
optimized out>) at /usr/src/sys/kern/kern_sx.c:926
#4 0xffffffff808d679d in _sx_slock (sx=<value optimized out>,
opts=<value optimized out>, file=<value optimized out>, line=<value
optimized out>) at sx.h:188
#5 0xffffffff818659ed in zap_get_leaf_byblk (zap=<value optimized out>,
blkid=9223372036854777145, tx=0x0, lt=RW_READER, lp=0xffffff824d2ce6e0)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zap.c:524
#6 0xffffffff81865e28 in zap_deref_leaf (zap=0xfffffe00c0af0e00,
h=<value optimized out>, tx=0x0, lt=RW_READER, lp=0xffffff824d2ce6e0) at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zap.c:585
#7 0xffffffff818660f7 in fzap_cursor_retrieve (zap=0xfffffe00c0af0e00,
zc=0xffffff824d2ce6d0, za=0xffffff824d2ce720) at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zap.c:1181
#8 0xffffffff8186abe5 in zap_cursor_retrieve (zc=0xffffff824d2ce6d0,
za=0xffffff824d2ce720) at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zap_micro.c:1290
#9 0xffffffff8189bbe8 in zfs_freebsd_readdir (ap=<value optimized out>)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:2550
#10 0xffffffff80db2e28 in VOP_READDIR_APV (vop=0xffffffff8191a6c0,
a=0xffffff824d2ce8e0) at vnode_if.c:1793
#11 0xffffffff809789f2 in kern_getdirentries (td=0xfffffe000c00c920,
fd=<value optimized out>, buf=0x80108e000 <Error reading address
0x80108e000: Bad address>, count=4096, basep=0xffffff824d2ce9b8) at
vnode_if.h:758
#12 0xffffffff80978cf3 in sys_getdirentries (td=<value optimized out>,
uap=0xffffff824d2cea70) at /usr/src/sys/kern/vfs_syscalls.c:4145
#13 0xffffffff80cb39aa in amd64_syscall (td=0xfffffe000c00c920,
traced=0) at subr_syscall.c:135
#14 0xffffffff80c9e107 in Xfast_syscall () at
/usr/src/sys/amd64/amd64/exception.S:391
#15 0x0000000800cffe7c in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb)
Thanks,
Kai
Am 19.02.2014 12:19, schrieb Andriy Gapon:
> on 19/02/2014 13:08 Kai Kockro said the following:
>> Hello Andriy,
>>
>>
>> ps -p 4883
>> PID TT STAT TIME COMMAND
>> 4883 1- D+ 0:00,00 ls -G -f
>>
>> procstat -kk 4883
>> PID TID COMM TDNAME KSTACK
>> 4883 100558 ls - mi_switch+0x194
>> sleepq_wait+0x42 _sx_slock_hard+0x3b0 _sx_slock+0x3d
>> zap_get_leaf_byblk+0xbd zap_deref_leaf+0x68 fzap_cursor_retrieve+0xe7
>> zap_cursor_retrieve+0x155 zfs_freebsd_readdir+0x2d8 VOP_READDIR_APV+0x78
>> kern_getdirentries+0x212 sys_getdirentries+0x23 amd64_syscall+0x5ea
>> Xfast_syscall+0xf7
> Can you please also obtain the following info?
> 1. Find out an inode number of the affected directory and its underlying ZFS
> filesystem, e.g. by using stat(1). Then execute `zdb -ddddddd <dataset> <inum>`.
>
> 2. Run 'kgdb', issue 'tid 100558' command and 'bt' command.
> Examine the stack trace and try to get as much information as possible from
> frames with _sx_slock and zap_get_leaf_byblk calls. For pointers please print
> objects that they point to (e.g. 'print *ptr').
> In particular I am interested in *sx, *zap, *l.
>
>> ps -p 5236
>> PID TT STAT TIME COMMAND
>> 5236 3- D+ 0:00,00 ls -G -la
>>
>> procstat -kk 5236
>> PID TID COMM TDNAME KSTACK
>> 5236 101104 ls - mi_switch+0x194
>> sleepq_wait+0x42 __lockmgr_args+0x9a0 vop_stdlock+0x39
>> VOP_LOCK1_APV+0x84 _vn_lock+0x47 vacl_get_acl+0x55
>> sys___acl_get_link+0x10d amd64_syscall+0x5ea Xfast_syscall+0xf7
>>
>>
>>
>> Best,
>> Kai
>>
>> Am 19.02.2014 11:53, schrieb Andriy Gapon:
>>> on 19/02/2014 12:31 Kai Kockro said the following:
>>>> Hello,
>>>>
>>>> i have an 9.2-STABLE r262153 installation on a T420s laptop. Its a ZROOT
>>>> on a INTEL SSDSA2BW160G3L drive.
>>>>
>>>> I have a folder ( skype profil ), which is not accessable anymore. I can
>>>> do ls, du and so on, but the states are in D+ after this.
>>>>
>>>> STRG + T shows:
>>>>
>>>> [user at host 11:01am] ~/.SkypeOLD/user_profil/>ls -f
>>>> load: 0.13 cmd: ls 4883 [] 2.76r 0.00u 0.00s 0% 2124k
>>>> load: 0.13 cmd: ls 4883 [] 3.50r 0.00u 0.00s 0% 2124k
>>>> load: 0.13 cmd: ls 4883 [] 3.92r 0.00u 0.00s 0% 2124k
>>>> load: 0.20 cmd: ls 4883 [] 4.13r 0.00u 0.00s 0% 2124k
>>>> load: 0.20 cmd: ls 4883 [] 4.31r 0.00u 0.00s 0% 2124k
>>>>
>>>> What can i do? Any debug infos needed?
>>> You can start with getting procstat -kk output for a stuck process.
>>>
>
--
Kai Kockro
Leitung Operations
myphotobook gmbh
Oranienstrasse 183
10999 Berlin
Tel.:+49 (0)30 616 508 100
Fax: +49 (0)30 616 508 200
Geschäftsführer: Vanessa Dill, Martin Lux
Amtsgericht Charlottenburg; HRB 94377
More information about the freebsd-fs
mailing list