zfs panic, solaris_assert

Fri May 31 20:23:33 UTC 2013

Hi,

On 12/16/2012 01:23 PM, Attila Nagy wrote:
> On 12/16/2012 01:21 PM, Andriy Gapon wrote:
>> on 15/12/2012 16:19 Attila Nagy said the following:
>>> Hi,
>>>
>>> Since running svn revision r243704 I get frequent panics:
>>> panic: solaris assert: sa.sa_magic == 0x2F505A (0x4f22a8ed == 0x2f505a),
>>> file:
>>> /data/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c,
>>> line: 597
>>> cpuid = 0
>>> KDB: stack backtrace:
>>> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
>>> kdb_backtrace() at kdb_backtrace+0x37
>>> panic() at panic+0x1ce
>>> assfail3() at assfail3+0x29
>>> zfs_space_delta_cb() at zfs_space_delta_cb+0xbe
>>> dmu_objset_userquota_get_ids() at dmu_objset_userquota_get_ids+0x142
>>> dnode_sync() at dnode_sync+0xc5
>>> dmu_objset_sync_dnodes() at dmu_objset_sync_dnodes+0x5d
>>> dmu_objset_sync() at dmu_objset_sync+0x17f
>>> dsl_pool_sync() at dsl_pool_sync+0xca
>>> spa_sync() at spa_sync+0x34a
>>> txg_sync_thread() at txg_sync_thread+0x139
>>> fork_exit() at fork_exit+0x11f
>>> fork_trampoline() at fork_trampoline+0xe
>>> --- trap 0, rip = 0, rsp = 0xffffff90231accf0, rbp = 0 ---
>>>
>>> I can't tell whether it's the data or the code. If the latter, is this fixed in
>>> later revisions?
>>> If it's the file system, what can I do with this?
>> Nobody's sure.
>> Do you have a crash dump?
>>
> Not yet.
Sorry, I've had better things to work on, but after a half year I can 
clearly say that something with that mega commit (maybe the v28 stuff, I 
can't really remember) broke this machine.
Since last December I was running r237433 (fallback, it had to work) 
without any problems. Then last week I did an upgrade to r250452.
After that, I've had some 5 or 6 crashes, two in a row today.

So now I have a more recent OS (reverted to r237433 again, but can 
easily switch versions) and a crash dump too.

The current panic message:
panic: solaris assert: sa.sa_magic == 0x2F505A (0x4e814a76 == 0x2f505a), 
file: 
/data/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c, 
line: 625
cpuid = 1
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a/frame 
0xffffffa68ce6b4d0
kdb_backtrace() at kdb_backtrace+0x37/frame 0xffffffa68ce6b590
panic() at panic+0x1ce/frame 0xffffffa68ce6b690
assfail3() at assfail3+0x29/frame 0xffffffa68ce6b6b0
zfs_space_delta_cb() at zfs_space_delta_cb+0xbe/frame 0xffffffa68ce6b700
dmu_objset_userquota_get_ids() at 
dmu_objset_userquota_get_ids+0x142/frame 0xffffffa68ce6b750
dnode_sync() at dnode_sync+0xc5/frame 0xffffffa68ce6b820
dmu_objset_sync_dnodes() at dmu_objset_sync_dnodes+0x5d/frame 
0xffffffa68ce6b850
dmu_objset_sync() at dmu_objset_sync+0x169/frame 0xffffffa68ce6b920
dsl_pool_sync() at dsl_pool_sync+0xca/frame 0xffffffa68ce6ba00
spa_sync() at spa_sync+0x3ba/frame 0xffffffa68ce6bad0
txg_sync_thread() at txg_sync_thread+0x139/frame 0xffffffa68ce6bbe0
fork_exit() at fork_exit+0x11f/frame 0xffffffa68ce6bc30
fork_trampoline() at fork_trampoline+0xe/frame 0xffffffa68ce6bc30
--- trap 0, rip = 0, rsp = 0xffffffa68ce6bcf0, rbp = 0 ---
Uptime: 10m6s

10 minutes of uptime. :-O

(kgdb) bt
#0  doadump (textdump=1) at /data/usr/src/sys/kern/kern_shutdown.c:266
#1  0xffffffff8092c754 in kern_reboot (howto=260) at 
/data/usr/src/sys/kern/kern_shutdown.c:449
#2  0xffffffff8092cc57 in panic (fmt=0x1 <Address 0x1 out of bounds>) at 
/data/usr/src/sys/kern/kern_shutdown.c:637
#3  0xffffffff81b64109 in vcmn_err (ce=0, fmt=0x0, adx=0x0) at 
/data/usr/src/sys/modules/opensolaris/../../cddl/compat/opensolaris/kern/opensolaris_cmn_err.c:29
#4  0xffffffff81abdc1e in zfs_freebsd_getattr (ap=<value optimized out>) 
at 
/data/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:2826
#5  0xffffffff81a44cd2 in dmu_tx_hold_object_impl 
(tx=0xfffffe080e38c6b0, os=<value optimized out>, object=<value 
optimized out>, type=<value optimized out>, arg1=<value optimized out>,
     arg2=<value optimized out>) at 
/data/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_tx.c:124
#6  0xffffffff81a4bd55 in dsl_dir_willuse_space_impl 
(dd=0xfffffe07d7100a00, space=-2164424915280, tx=0x5)
     at 
/data/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_dir.c:919
#7  0xffffffff81a4305d in dmu_objset_snapshot (fsname=0x0, snapname=0x1 
<Address 0x1 out of bounds>, tag=0xffffffff8141dc40 "", 
props=0xfffffe0049be1de0, recursive=1237196368, temporary=1237196256,
     cleanup_fd=-1931036464) at 
/data/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_objset.c:1018
#8  0xffffffff81a431e9 in dmu_objset_byteswap (buf=0x0, 
size=18446744071589605469) at 
/data/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_objset.c:250
#9  0xffffffff81a5930a in dmu_zfetch_dofetch (zf=0x0, zs=0x0) at 
/data/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_zfetch.c:200
#10 0xffffffff81a6b02a in spa_load (spa=0x0, state=10031132, 
type=3608152576, mosconfig=1237195776) at 
/data/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c:2077
#11 0xffffffff81a7d109 in vdev_free (vd=0xfffffe006f580800) at 
/data/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev.c:601
#12 0xffffffff808f971f in fork_exit (callout=0xffffffff81a7cfd0 
<vdev_top_transfer+336>, arg=0xfffffe006f580800, 
frame=0xffffffa68ce6bc40) at /data/usr/src/sys/kern/kern_fork.c:988
#13 0xffffffff80c8ad1e in fork_trampoline () at 
/data/usr/src/sys/amd64/amd64/exception.S:602
#14 0x0000000000000000 in ?? ()

I can do any non-destructive (for the data and the operation of the 
machine) stuff, if needed.

Thanks,