Kernel panic zio.c, line: 270 FreeBSD 10.2 (or 10.3)
DemIS
demis at yandex.ru
Mon Mar 14 11:12:22 UTC 2016
Hi, Steven,
more information for you:
I wrote a script component.
He gets all the directory inode and checks them through the program zdb -A -dddddd inode.
And this is not without causing the server is restarted, only the crash zdb program.
Over the past few weeks (so much time working script, one inode for about three seconds) unable to verify 163272 folder.
I got 55 inode folders with errors, an example of the error:
zdb -A -dddddd hdd/usr/wf 7962880
Dataset hdd/usr/wf [ZPL], ID 44, cr_txg 25, 3.82T, 2548316 objects, rootbp DVA[0]=<0:621dba8000:3000> DVA[1]=<0:3a4f9aa0000:3000> [L0 DMU objset] fletcher4 uncompressed LE contiguous unique double size=800L/800P birth=19128707L/19128707P fill=2548316 cksum=d2d2f7fd9:fb64a627f1e:bb115b7f9840d:672c8b8b245ae81
Object lvl iblk dblk dsize lsize %full type
7962880 1 16K 512 28.5K 512 100.00 ZFS directory (K=inherit) (Z=inherit)
304 bonus System attributes
dnode flags: USED_BYTES USERUSED_ACCOUNTED SPILL_BLKPTR
dnode maxblkid: 0
path /_Студия_ГП/МО_Отрадненское ГП/04-ГПМО_ПИ-02/07-Другое
uid 10010
gid 10001
atime Mon Mar 14 13:33:12 2016
mtime Fri May 22 12:52:21 2015
ctime Thu Dec 10 10:32:40 2015
crtime Thu Dec 10 10:32:40 2015
gen 17690948
mode 40000
size 2
parent 7962875
links 2
pflags 40800000042
Assertion failed: db->db.db_size >= dn->dn_bonuslen (0x0 >= 0x130), file /usr/src/cddl/lib/libzpool/../../../sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c, line 339.
Assertion failed: (!BP_IS_EMBEDDED(bp) || BPE_GET_ETYPE(bp) == BP_EMBEDDED_TYPE_DATA), file /usr/src/cddl/lib/libzpool/../../../sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c, line 3293.
Assertion failed: size > 0 (0x0 > 0x0), file /usr/src/cddl/lib/libzpool/../../../sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c, line 1536.
Assertion failed: (bytes > 0), file /usr/src/cddl/lib/libzpool/../../../sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c, line 2789.
Assertion failed: c < (1ULL << 24) >> 9 (0x7fffffffffffff < 0x8000), file /usr/src/cddl/lib/libzpool/../../../sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c, line 270.
Bus error (core dumped)
I plan to do the same for node files.
But:
1. This is a very very long time.
2. every time when there is a crash of the entire system has to run the script again, because log processing greatly damaged.
And almost no chance that the script files for my time to work through.
Working copies of damaged files and folders I have in backups.
But the problem is that I can not replace a broken folder, because rm folder or rm file leads to a crash.
14.03.2016, 01:27, "Steven Hartland" <killing at multiplay.co.uk>:
> Thanks DemIS, size = 0 is definitely bad however I don't see bp listed
> from frame #8.
>
> From the debugger:
> > frame 8
> > print bp
>
> On 13/03/2016 21:02, DemIS wrote:
>> Hi Steven ,
>>
>> Finally I was able to get the data that you have requested.
>> The following quote from the dump to
>> setfacl -b -m <someAccount> /hdd/usr/wf/some/long/path
>>
>> kgdb /usr/obj/usr/src/sys/TEO/kernel.debug /usr/crash/vmcore.last
>> GNU gdb 6.1.1 [FreeBSD]
>> Copyright 2004 Free Software Foundation, Inc.
>> GDB is free software, covered by the GNU General Public License, and you are
>> welcome to change it and/or distribute copies of it under certain conditions.
>> Type "show copying" to see the conditions.
>> There is absolutely no warranty for GDB. Type "show warranty" for details.
>> This GDB was configured as "amd64-marcel-freebsd"...
>>
>> Unread portion of the kernel message buffer:
>> panic: solaris assert: c < (1ULL << 24) >> 9 (0x7fffffffffffff < 0x8000), file: /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c, line: 270
>> cpuid = 2
>> KDB: stack backtrace:
>> db_trace_self_wrapper() at db_trace_self_wrapper+0x2e/frame 0xfffffe0677ebfbe0
>> kdb_backtrace() at kdb_backtrace+0x58/frame 0xfffffe0677ebfc90
>> vpanic() at vpanic+0x1df/frame 0xfffffe0677ebfd00
>> vpanic() at vpanic/frame 0xfffffe0677ebfd60
>> zfs_kmem_alloc() at zfs_kmem_alloc/frame 0xfffffe0677ebfdb0
>> zio_buf_alloc() at zio_buf_alloc+0x84/frame 0xfffffe0677ebfdf0
>> arc_get_data_buf() at arc_get_data_buf+0x7a/frame 0xfffffe0677ebfe60
>> arc_buf_alloc() at arc_buf_alloc+0xe0/frame 0xfffffe0677ebfeb0
>> arc_read() at arc_read+0x65d/frame 0xfffffe0677ec0040
>> dbuf_read_impl() at dbuf_read_impl+0x4af/frame 0xfffffe0677ec00e0
>> dbuf_read() at dbuf_read+0x33b/frame 0xfffffe0677ec0190
>> dmu_spill_hold_by_dnode() at dmu_spill_hold_by_dnode+0xf7/frame 0xfffffe0677ec01e0
>> dmu_spill_hold_existing() at dmu_spill_hold_existing+0x16b/frame 0xfffffe0677ec0240
>> sa_get_spill() at sa_get_spill+0x3c/frame 0xfffffe0677ec0260
>> sa_modify_attrs() at sa_modify_attrs+0x118/frame 0xfffffe0677ec0330
>> sa_attr_op() at sa_attr_op+0x9db/frame 0xfffffe0677ec03a0
>> sa_bulk_update_impl() at sa_bulk_update_impl+0x74/frame 0xfffffe0677ec03f0
>> sa_bulk_update() at sa_bulk_update+0x45/frame 0xfffffe0677ec0430
>> zfs_aclset_common() at zfs_aclset_common+0x95f/frame 0xfffffe0677ec0620
>> zfs_setacl() at zfs_setacl+0x451/frame 0xfffffe0677ec06c0
>> zfs_setsecattr() at zfs_setsecattr+0x10d/frame 0xfffffe0677ec0720
>> zfs_freebsd_setacl() at zfs_freebsd_setacl+0x15c/frame 0xfffffe0677ec0790
>> VOP_SETACL_APV() at VOP_SETACL_APV+0x174/frame 0xfffffe0677ec07f0
>> VOP_SETACL() at VOP_SETACL+0x5a/frame 0xfffffe0677ec0860
>> vacl_set_acl() at vacl_set_acl+0xf1/frame 0xfffffe0677ec08c0
>> sys___acl_set_file() at sys___acl_set_file+0x91/frame 0xfffffe0677ec09d0
>> syscallenter() at syscallenter+0x754/frame 0xfffffe0677ec0af0
>> amd64_syscall() at amd64_syscall+0x1f/frame 0xfffffe0677ec0bf0
>> Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0677ec0bf0
>> --- syscall (348, FreeBSD ELF64, sys___acl_set_file), rip = 0x80092a17a, rsp = 0x7fffffffe2e8, rbp = 0x7fffffffe7c0 ---
>> Uptime: 2d11h24m58s
>> Dumping 5431 out of 24533 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%
>>
>> Reading symbols from /boot/kernel/if_lagg.ko.symbols...done.
>> Loaded symbols for /boot/kernel/if_lagg.ko.symbols
>> Reading symbols from /boot/kernel/aio.ko.symbols...done.
>> Loaded symbols for /boot/kernel/aio.ko.symbols
>> Reading symbols from /boot/kernel/ichsmb.ko.symbols...done.
>> Loaded symbols for /boot/kernel/ichsmb.ko.symbols
>> Reading symbols from /boot/kernel/ipmi.ko.symbols...done.
>> Loaded symbols for /boot/kernel/ipmi.ko.symbols
>> Reading symbols from /boot/kernel/zfs.ko.symbols...done.
>> Loaded symbols for /boot/kernel/zfs.ko.symbols
>> Reading symbols from /boot/kernel/opensolaris.ko.symbols...done.
>> Loaded symbols for /boot/kernel/opensolaris.ko.symbols
>> Reading symbols from /boot/kernel/ums.ko.symbols...done.
>> Loaded symbols for /boot/kernel/ums.ko.symbols
>> Reading symbols from /boot/kernel/smbfs.ko.symbols...done.
>> Loaded symbols for /boot/kernel/smbfs.ko.symbols
>> Reading symbols from /boot/kernel/libiconv.ko.symbols...done.
>> Loaded symbols for /boot/kernel/libiconv.ko.symbols
>> Reading symbols from /boot/kernel/libmchain.ko.symbols...done.
>> Loaded symbols for /boot/kernel/libmchain.ko.symbols
>> #0 doadump (textdump=1) at /usr/src/sys/kern/kern_shutdown.c:263
>> 263 dumptid = curthread->td_tid;
>> (kgdb) bt full
>> #0 doadump (textdump=1) at /usr/src/sys/kern/kern_shutdown.c:263
>> coredump = 1
>> #1 0xffffffff80e21a6a in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:451
>> first_buf_printf = 1
>> #2 0xffffffff80e2240c in vpanic (fmt=0xffffffff8268f154 "solaris assert: %s (0x%jx %s 0x%jx), file: %s, line: %d", ap=0xfffffe0677ebfd40)
>> at /usr/src/sys/kern/kern_shutdown.c:758
>> buf = "solaris assert: c < (1ULL << 24) >> 9 (0x7", 'f' <repeats 13 times>, " < 0x8000), file: /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c, line: 270", '\0' <repeats 92 times>
>> other_cpus = {__bits = {65531}}
>> td = (struct thread *) 0xfffff800713124a0
>> bootopt = 260
>> newpanic = 1
>> #3 0xffffffff80e221d0 in panic (fmt=0xffffffff8268f154 "solaris assert: %s (0x%jx %s 0x%jx), file: %s, line: %d")
>> at /usr/src/sys/kern/kern_shutdown.c:687
>> ap = {{gp_offset = 48, fp_offset = 48, overflow_arg_area = 0xfffffe0677ebfd78, reg_save_area = 0xfffffe0677ebfd10}}
>> #4 0xffffffff8268e470 in assfail3 (a=0xffffffff825cbcb9 "c < (1ULL << 24) >> 9", lv=36028797018963967, op=0xffffffff825cbccf "<", rv=32768,
>> f=0xffffffff825cbcd1 "/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c", l=270)
>> at /usr/src/sys/modules/opensolaris/../../cddl/compat/opensolaris/kern/opensolaris_cmn_err.c:91
>> No locals.
>> #5 0xffffffff8250e414 in zio_buf_alloc (size=0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:270
>> __left = 36028797018963967
>> __right = 32768
>> c = 36028797018963967
>> flags = 0
>> #6 0xffffffff8243dc1a in arc_get_data_buf (buf=0xfffff804795ddc18)
>> at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:2898
>> state = (arc_state_t *) 0xffffffff825e8700
>> size = 0
>> type = ARC_BUFC_METADATA
>> #7 0xffffffff8243db60 in arc_buf_alloc (spa=0xfffff800719d1000, size=0, tag=0xfffff803b1e8de00, type=ARC_BUFC_METADATA)
>> at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:1551
>> hdr = (arc_buf_hdr_t *) 0xfffff8025cc22288
>> buf = (arc_buf_t *) 0xfffff804795ddc18
>> #8 0xffffffff824419bd in arc_read (pio=0xfffff80071def398, spa=0xfffff800719d1000, bp=0xfffffe001589e180, done=0xffffffff82455380 <dbuf_read_done>,
>> private=0xfffff803b1e8de00, priority=ZIO_PRIORITY_SYNC_READ, zio_flags=128, arc_flags=0xfffffe0677ec0094, zb=0xfffffe0677ec0098)
>> at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:3389
>> exists = (arc_buf_hdr_t *) 0x0
>> type = ARC_BUFC_METADATA
>> vd = (vdev_t *) 0x0
>> devw = 0
>> b_asize = 0
>> size = 0
>> acb = (arc_callback_t *) 0xfffff800713124a0
>> addr = 0
>> b_compress = ZIO_COMPRESS_OFF
>> sdt_sdt___arc__hit = {{version = 1287817688, prov = 0x30c48348fffff834, probe_entry = {tqe_next = 0x2e6666666666c35d, tqe_prev = 0x841f0f},
>> argtype_list = {tqh_first = 0x20ec8348e5894855, tqh_last = 0xf0758948f87d8948},
>> mod = 0xf87d8b48e8558948 <Address 0xf87d8b48e8558948 out of bounds>, func = 0xe8558b48f0758b48 <Address 0xe8558b48f0758b48 out of bounds>,
>> name = 0x12b8c189c031 <Address 0x12b8c189c031 out of bounds>, id = 5531134888965130240, n_args = -387955317, sdtp_lf = 0x9066c35d20c48348}}
>> __set_sdt_probes_set_sym_sdt_sdt___arc__hit = (const void * const) 0x58f0f000000
>> sdta_sdt___arc__hit0 = {{ndx = -443987883, type = 0x4c984d894c000001 <Address 0x4c984d894c000001 out of bounds>,
>> xtype = 0x48884d8948904589 <Address 0x48884d8948904589 out of bounds>, argtype_entry = {tqe_next = 0xb8d04d8d48805589, tqe_prev = 0xba4100000001},
>> probe = 0x7589f87d89480000}}
>> ---Type <return> to continue, or q <return> to quit---
>> __set_sdt_argtypes_set_sym_sdta_sdt___arc__hit0 = (const void * const) 0x1518e0f000000
>> sdt_sdt___arc__miss = {{version = -193623052, prov = 0xfe810000, probe_entry = {tqe_next = 0x41db8445c3950f41, tqe_prev = 0x45c7c44589c2440f},
>> argtype_list = {tqh_first = 0x958d4800000000c0, tqh_last = 0x10518948ffffff70}, mod = 0x851894810558d48 <Address 0x851894810558d48 out of bounds>,
>> func = 0xc7000000300441c7 <Address 0xc7000000300441c7 out of bounds>, name = 0x8100b00000001001 <Address 0x8100b00000001001 out of bounds>,
>> id = -8824803469832437635, n_args = -145, sdtp_lf = 0x4d8bd0458d480000}}
>> __set_sdt_probes_set_sym_sdt_sdt___arc__miss = (const void * const) 0x2b800000174e900
>> sdta_sdt___arc__miss0 = {{ndx = 687440336, type = 0x8d89ffffff608589 <Address 0x8d89ffffff608589 out of bounds>,
>> xtype = 0x2b870fffffff5c <Address 0x2b870fffffff5c out of bounds>, argtype_entry = {tqe_next = 0xffff60858b480000, tqe_prev = 0x5c958b10488b48ff},
>> probe = 0x148f26348ffffff}}
>> __set_sdt_argtypes_set_sym_sdta_sdt___arc__miss0 = (const void * const) 0x8be04589e4458b00
>> sdta_sdt___arc__miss1 = {{ndx = 146964977, type = 0xffffff508d894810 <Address 0xffffff508d894810 out of bounds>,
>> xtype = 0x858b4800000020e9 <Address 0x858b4800000020e9 out of bounds>, argtype_entry = {tqe_next = 0x8488b48ffffff60, tqe_prev = 0x8c18148ca8948},
>> probe = 0x8948084889480000}}
>> __set_sdt_argtypes_set_sym_sdta_sdt___arc__miss1 = (const void * const) 0x1f8d0ff4453be045
>> sdta_sdt___arc__miss2 = {{ndx = -44907, type = 0x48008b48ffffff50 <Address 0x48008b48ffffff50 out of bounds>,
>> xtype = 0x3d48c84589 <Address 0x3d48c84589 out of bounds>, argtype_entry = {tqe_next = 0xff6f8d88c1950f00, tqe_prev = 0xffffff6f858affff},
>> probe = 0x5850f01a8}}
>> __set_sdt_argtypes_set_sym_sdta_sdt___arc__miss2 = (const void * const) 0x452be0458b000001
>> sdta_sdt___arc__miss3 = {{ndx = 186601, type = 0x28f981d04d8bd0 <Address 0x28f981d04d8bd0 out of bounds>,
>> xtype = 0xffff488589480000 <Address 0xffff488589480000 out of bounds>, argtype_entry = {tqe_next = 0xfffffff448d89ff,
>> tqe_prev = 0x858b480000002b87}, probe = 0x10488b48ffffff48}}
>> __set_sdt_argtypes_set_sym_sdta_sdt___arc__miss3 = (const void * const) 0xdc7d81dc4589e4
>> sdt_sdt___l2arc__hit = {{version = -12282485, prov = 0x8c281f10148f2, probe_entry = {tqe_next = 0x388d894810890000, tqe_prev = 0x20e9ffffff},
>> argtype_list = {tqh_first = 0x48ffffff48858b48, tqh_last = 0x8148ca894808488b},
>> mod = 0x48894800000008c1 <Address 0x48894800000008c1 out of bounds>, func = 0xffffff3895894808 <Address 0xffffff3895894808 out of bounds>,
>> name = 0x8bffffff38858b48 <Address 0x8bffffff38858b48 out of bounds>, id = 5587026407265569032, n_args = -1982953028, sdtp_lf = 0x89ffffff309589ff}}
>> __set_sdt_probes_set_sym_sdt_sdt___l2arc__hit = (const void * const) 0x4589f44d8b000000
>> sdta_sdt___l2arc__hit0 = {{ndx = -54091, type = 0xec03fcb8000002 <Address 0xec03fcb8000002 out of bounds>,
>> xtype = 0xa30fffffff348d8b <Address 0xa30fffffff348d8b out of bounds>, argtype_entry = {tqe_next = 0xb800000058820fc8,
>> tqe_prev = 0xff348d8b0713fc00}, probe = 0xdb820fc8a30fffff}}
>> __set_sdt_argtypes_set_sym_sdta_sdt___l2arc__hit0 = (const void * const) 0xf48c0f000000
>> sdt_sdt___l2arc__read = {{version = -1962934272, prov = 0xff28858901e883ff, probe_entry = {tqe_next = 0x5840fffff,
>> tqe_prev = 0x7d8b48000001d3e9}, argtype_list = {tqh_first = 0xbc558bc8758b48f8, tqh_last = 0x208d8948c189c031},
>> mod = 0xff20858b4cffffff <Address 0xff20858b4cffffff out of bounds>, func = 0x89fffff5c1e8ffff <Address 0x89fffff5c1e8ffff out of bounds>,
>> name = 0x48000001b0e9c045 <Address 0x48000001b0e9c045 out of bounds>, id = -467863854793079411, n_args = 40, sdtp_lf = 0xffff148d89ffffff}}
>> __set_sdt_probes_set_sym_sdt_sdt___l2arc__read = (const void * const) 0xf7c04d8b99c889c0
>> sdta_sdt___l2arc__read0 = {{ndx = 730271743, type = 0x8b48ffffff18858b <Address 0x8b48ffffff18858b out of bounds>,
>> xtype = 0xffffff14958b1048 <Address 0xffffff14958b1048 out of bounds>, argtype_entry = {tqe_next = 0xc281f10148f26348,
>> tqe_prev = 0x8948108900000008}, probe = 0x20e9ffffff088d}}
>> __set_sdt_argtypes_set_sym_sdta_sdt___l2arc__read0 = (const void * const) 0x4d8b48e8458b4800
>> sdta_sdt___l2arc__read1 = {{ndx = -1958215680, type = 0xca894808488b48ff <Address 0xca894808488b48ff out of bounds>,
>> xtype = 0x4800000008c18148 <Address 0x4800000008c18148 out of bounds>, argtype_entry = {tqe_next = 0xff08958948084889,
>> tqe_prev = 0xffff08858b48ffff}, probe = 0xb0458948008b48ff}}
>> __set_sdt_argtypes_set_sym_sdta_sdt___l2arc__read1 = (const void * const) 0xf055af0fdc558bf8
>> sdt_sdt___l2arc__miss = {{version = -125990072, prov = 0x31b0458b4cbc558b, probe_entry = {tqe_next = 0x4589fffff52ae8c9,
>> tqe_prev = 0x8d4800000119e9c0}, argtype_list = {tqh_first = 0x28f981d04d8bd045, tqh_last = 0xff00858948000000},
>> mod = 0xfffffefc8d89ffff <Address 0xfffffefc8d89ffff out of bounds>, func = 0x8b480000002b870f <Address 0x8b480000002b870f out of bounds>,
>> name = 0x488b48ffffff0085 <Address 0x488b48ffffff0085 out of bounds>, id = 5260204360416463632, n_args = 21557859, sdtp_lf = 0x8d89481089000000}}
>> __set_sdt_probes_set_sym_sdt_sdt___l2arc__miss = (const void * const) 0xe47d81e44589f9
>> sdta_sdt___l2arc__miss0 = {{ndx = -272, type = 0xffffff00858b4800 <Address 0xffffff00858b4800 out of bounds>,
>> xtype = 0x48ca894808488b48 <Address 0x48ca894808488b48 out of bounds>, argtype_entry = {tqe_next = 0x894800000008c181,
>> tqe_prev = 0xfffef09589480848}, probe = 0xfffffef0858b48ff}}
>> __set_sdt_argtypes_set_sym_sdta_sdt___l2arc__miss0 = (const void * const) 0x8b48f10148f26348
>> ---Type <return> to continue, or q <return> to quit---
>> sdt_sdt___l2arc__miss = {{version = -800223928, prov = 0xfa81d0558bb04589, probe_entry = {tqe_next = 0xe88d894800000028,
>> tqe_prev = 0xfffee49589fffffe}, argtype_list = {tqh_first = 0x480000002b870fff, tqh_last = 0x8b48fffffee8858b},
>> mod = 0xfffffee4958b1048 <Address 0xfffffee4958b1048 out of bounds>, func = 0xc281f10148f26348 <Address 0xc281f10148f26348 out of bounds>,
>> name = 0x8948108900000008 <Address 0x8948108900000008 out of bounds>, id = 9264484975564941, n_args = -1958215680, sdtp_lf = 0xca894808488b48ff}}
>> __set_sdt_probes_set_sym_sdt_sdt___l2arc__miss = (const void * const) 0x1518e0f000000
>> sdta_sdt___l2arc__miss0 = {{ndx = 146899272, type = 0xfed8958948084889 <Address 0xfed8958948084889 out of bounds>,
>> xtype = 0xfffed8858b48ffff <Address 0xfffed8858b48ffff out of bounds>, argtype_entry = {tqe_next = 0xa8458948008b48ff,
>> tqe_prev = 0xc8758b48f87d8b48}, probe = 0x4ca84d8b48bc558b}}
>> __set_sdt_argtypes_set_sym_sdta_sdt___l2arc__miss0 = (const void * const) 0xe45503dc558bf875
>> hdr = (arc_buf_hdr_t *) 0x0
>> buf = (arc_buf_t *) 0x0
>> hash_lock = (kmutex_t *) 0x0
>> rzio = (zio_t *) 0xfffffe0677ebfff0
>> guid = 18385013180861733512
>> #9 0xffffffff8244f7df in dbuf_read_impl (db=0xfffff803b1e8de00, zio=0xfffff80071def398, flags=0xfffffe0677ec0174)
>> at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:573
>> dn = (dnode_t *) 0xfffff805c8be5000
>> zb = {zb_objset = 44, zb_object = 7962880, zb_level = 0, zb_blkid = 18446744073709551614}
>> aflags = 100
>> #10 0xffffffff8244ef8b in dbuf_read (db=0xfffff803b1e8de00, zio=0xfffff80071def398, flags=6)
>> at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:621
>> spa = (spa_t *) 0xfffff800719d1000
>> sdt_sdt___blocked__read = {{version = 549749576, prov = 0x841f0f, probe_entry = {tqe_next = 0x20ec8348e5894855, tqe_prev = 0xf0758948f87d8948},
>> argtype_list = {tqh_first = 0xf0758b48f87d8b48, tqh_last = 0x6b8c289c031}, mod = 0x48c689e875894800 <Address 0x48c689e875894800 out of bounds>,
>> func = 0xfffffc60e8e84d8b <Address 0xfffffc60e8e84d8b out of bounds>, name = 0x2e66c35d20c48348 <Address 0x2e66c35d20c48348 out of bounds>,
>> id = 8658703, n_args = -443987883, sdtp_lf = 0xf0758948f87d8948}}
>> __set_sdt_probes_set_sym_sdt_sdt___blocked__read = (const void * const) 0xd089b44589f0558b
>> sdta_sdt___blocked__read0 = {{ndx = -125990072, type = 0x7b8c289c031 <Address 0x7b8c289c031 out of bounds>,
>> xtype = 0x48c689e875894800 <Address 0x48c689e875894800 out of bounds>, argtype_entry = {tqe_next = 0xfffffc20e8e84d8b,
>> tqe_prev = 0x2e66c35d20c48348}, probe = 0x841f0f}}
>> __set_sdt_argtypes_set_sym_sdta_sdt___blocked__read0 = (const void * const) 0x4c28148c28948
>> sdta_sdt___blocked__read1 = {{ndx = -443987883, type = 0xf0758948f87d8948 <Address 0xf0758948f87d8948 out of bounds>,
>> xtype = 0xf0758b48f87d8b48 <Address 0xf0758b48f87d8b48 out of bounds>, argtype_entry = {tqe_next = 0x8b8c289c031, tqe_prev = 0x48c689e875894800},
>> probe = 0xfffffbe0e8e84d8b}}
>> __set_sdt_argtypes_set_sym_sdta_sdt___blocked__read1 = (const void * const) 0x889c85589480000
>> err = 0
>> havepzio = 0
>> prefetch = 1
>> dn = (dnode_t *) 0xfffff805c8be5000
>> #11 0xffffffff8245a777 in dmu_spill_hold_by_dnode (dn=0xfffff805c8be5000, flags=6, tag=0x0, dbp=0xfffff802f48dfa28)
>> at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu.c:333
>> db = (dmu_buf_impl_t *) 0xfffff803b1e8de00
>> err = -512
>> #12 0xffffffff8245a91b in dmu_spill_hold_existing (bonus=0xfffff80052609620, tag=0x0, dbp=0xfffff802f48dfa28)
>> at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu.c:359
>> db = (dmu_buf_impl_t *) 0xfffff80052609620
>> dn = (dnode_t *) 0xfffff805c8be5000
>> err = 1
>> #13 0xffffffff824b140c in sa_get_spill (hdl=0xfffff802f48dfa00) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/sa.c:310
>> rc = 0
>> #14 0xffffffff824b1658 in sa_modify_attrs (hdl=0xfffff802f48dfa00, newattr=19, action=SA_REPLACE, locator=0xffffffff82520d50 <zfs_acl_data_locator>,
>> datastart=0xfffffe0677ec05c8, buflen=152, tx=0xfffff8034ef07800)
>> at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/sa.c:1681
>> ---Type <return> to continue, or q <return> to quit---
>> sa = (sa_os_t *) 0xfffff80071e5da00
>> db = (dmu_buf_impl_t *) 0xfffff80052609620
>> dn = (dnode_t *) 0xfffff805c8be5000
>> attr_desc = (sa_bulk_attr_t *) 0xfffff803c1d2a000
>> old_data = {0xfffff80122383800, 0xfffff8034ef07800}
>> bonus_attr_count = 14
>> bonus_data_size = 320
>> spill_data_size = 0
>> spill_attr_count = 0
>> error = -2045
>> length = 49618
>> i = 0
>> j = 0
>> k = 2
>> length_idx = 16691240
>> hdr = (sa_hdr_phys_t *) 0xfffff80052600200
>> idx_tab = (sa_idx_tab_t *) 0xffffffff825bb2ea
>> attr_count = 0
>> count = -1367343104
>> #15 0xffffffff824b12fb in sa_attr_op (hdl=0xfffff802f48dfa00, bulk=0xfffffe0677ec04f0, count=5, data_op=SA_UPDATE, tx=0xfffff8034ef07800)
>> at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/sa.c:392
>> sa = (sa_os_t *) 0xfffff80071e5da00
>> i = 3
>> error = 0
>> buftypes = SA_BONUS
>> #16 0xffffffff824b5a04 in sa_bulk_update_impl (hdl=0xfffff802f48dfa00, bulk=0xfffffe0677ec04f0, count=5, tx=0xfffff8034ef07800)
>> at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/sa.c:1790
>> error = 1888
>> sa = (sa_os_t *) 0xfffff80071e5da00
>> bonustype = DMU_OT_SA
>> #17 0xffffffff824b5ca5 in sa_bulk_update (hdl=0xfffff802f48dfa00, attrs=0xfffffe0677ec04f0, count=5, tx=0xfffff8034ef07800)
>> at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/sa.c:1889
>> error = 1
>> #18 0xffffffff82521b5f in zfs_aclset_common (zp=0xfffff803b27e85c0, aclp=0xfffff8001ebcfa00, cr=0xfffff804455e6e00, tx=0xfffff8034ef07800)
>> at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_acl.c:1329
>> error = -2045
>> zfsvfs = (zfsvfs_t *) 0xfffff80086ee2000
>> otype = DMU_OT_ACL
>> locate = {cb_aclp = 0xfffff8001ebcfa00, cb_acl_node = 0x0}
>> mode = 16893
>> bulk = {{sa_data = 0xfffffe0677ec05c0, sa_data_func = 0, sa_length = 8, sa_attr = 5, sa_addr = 0xfffff80052600208, sa_buftype = 1,
>> sa_size = 8}, {sa_data = 0xfffff803b27e86c8, sa_data_func = 0, sa_length = 8, sa_attr = 11, sa_addr = 0xfffff80052600238, sa_buftype = 1,
>> sa_size = 8}, {sa_data = 0xfffffe0677ec04e0, sa_data_func = 0, sa_length = 16, sa_attr = 2, sa_addr = 0xfffff80052600260, sa_buftype = 1,
>> sa_size = 16}, {sa_data = 0xfffffe0677ec05c8, sa_data_func = 0xffffffff82520d50 <zfs_acl_data_locator>, sa_length = 152, sa_attr = 19,
>> sa_addr = 0xfffff80052600290, sa_buftype = 1, sa_size = 160}, {sa_data = 0xfffff8001ebcfa00, sa_data_func = 0, sa_length = 8, sa_attr = 16,
>> sa_addr = 0xfffffe0008161250, sa_buftype = 0, sa_size = 48}}
>> ctime = {1457872316, 787888506}
>> count = 5
>> #19 0xffffffff82524391 in zfs_setacl (zp=0xfffff803b27e85c0, vsecp=0xfffffe0677ec0748, skipaclchk=0, cr=0xfffff804455e6e00)
>> at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_acl.c:2010
>> zfsvfs = (zfsvfs_t *) 0xfffff80086ee2000
>> zilog = (zilog_t *) 0xfffff80086ebd000
>> mask = 16
>> ---Type <return> to continue, or q <return> to quit---
>> tx = (dmu_tx_t *) 0xfffff8034ef07800
>> error = 0
>> aclp = (zfs_acl_t *) 0xfffff8001ebcfa00
>> fuidp = (zfs_fuid_info_t *) 0x0
>> fuid_dirtied = 0
>> acl_obj = 0
>> #20 0xffffffff8254e33d in zfs_setsecattr (vp=0xfffff8034dbfa000, vsecp=0xfffffe0677ec0748, flag=0, cr=0xfffff804455e6e00, ct=0x0)
>> at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:5332
>> zp = (znode_t *) 0xfffff803b27e85c0
>> zfsvfs = (zfsvfs_t *) 0xfffff80086ee2000
>> error = 2
>> skipaclchk = 0
>> zilog = (zilog_t *) 0xfffff80086ebd000
>> #21 0xffffffff8254ff5c in zfs_freebsd_setacl (ap=0xfffffe0677ec0808)
>> at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:7076
>> error = 0
>> vsecattr = {vsa_mask = 16, vsa_aclcnt = 11, vsa_aclentp = 0xfffff804bd725e00, vsa_dfaclcnt = 2011957208, vsa_dfaclentp = 0xfffffe0677ec07c0,
>> vsa_aclentsz = 132, vsa_aclflags = 2}
>> aclbsize = 132
>> aaclp = (aclent_t *) 0xfffff804bd725e00
>> #22 0xffffffff816db594 in VOP_SETACL_APV (vop=0xffffffff825de1a0, a=0xfffffe0677ec0808) at vnode_if.c:2993
>> mp__ = (struct mount *) 0xfffff80086df5cc0
>> _enable_stops = 0
>> rc = -1
>> #23 0xffffffff80f0ba5a in VOP_SETACL (vp=0xfffff8034dbfa000, type=4, aclp=0xfffff800612a1000, cred=0xfffff804455e6e00, td=0xfffff800713124a0)
>> at vnode_if.h:1253
>> a = {a_gen = {a_desc = 0xffffffff81d77eb8}, a_vp = 0xfffff8034dbfa000, a_type = 4, a_aclp = 0xfffff800612a1000, a_cred = 0xfffff804455e6e00,
>> a_td = 0xfffff800713124a0}
>> #24 0xffffffff80f0aea1 in vacl_set_acl (td=0xfffff800713124a0, vp=0xfffff8034dbfa000, type=4, aclp=0x80100a000) at /usr/src/sys/kern/vfs_acl.c:232
>> inkernelacl = (struct acl *) 0xfffff800612a1000
>> mp = (struct mount *) 0xfffff80086df5cc0
>> error = 0
>> #25 0xffffffff80f0ad81 in sys___acl_set_file (td=0xfffff800713124a0, uap=0xfffffe0677ec0b98) at /usr/src/sys/kern/vfs_acl.c:375
>> nd = {ni_dirp = 0x7fffffffecd2 <Address 0x7fffffffecd2 out of bounds>, ni_segflg = UIO_USERSPACE, ni_rightsneeded = {cr_rights = {
>> 144115188075855872, 288230376151711744}}, ni_startdir = 0x0, ni_rootdir = 0xfffff800714cfb10, ni_topdir = 0x0, ni_dirfd = -100,
>> ni_strictrelative = 0, ni_filecaps = {fc_rights = {cr_rights = {0, 0}}, fc_ioctls = 0x0, fc_nioctls = -1, fc_fcntls = 0}, ni_vp = 0xfffff8034dbfa000,
>> ni_dvp = 0xfffff805c678d000, ni_pathlen = 1, ni_next = 0xfffff80086ddcc65 "", ni_loopcnt = 0, ni_cnd = {cn_nameiop = 0, cn_flags = 49216,
>> cn_thread = 0xfffff800713124a0, cn_cred = 0xfffff804455e6e00, cn_lkflags = 2097152,
>> cn_pnbuf = 0xfffff80086ddcc00 "/hdd/usr/wf/_С-\202-\203ди-\217_-\223-\237/-\234-\236_-\236-\202-\200аднен-\201кое -\223-\237/04--\223-\237-\234-\236_-\237-\230-02/07--\224-\200-\203гое", cn_nameptr = 0xfffff80086ddcc56 "07--\224-\200-\203гое", cn_namelen = 15, cn_consume = 0}}
>> error = 0
>> #26 0xffffffff814e32f4 in syscallenter (td=0xfffff800713124a0, sa=0xfffffe0677ec0b88) at subr_syscall.c:134
>> p = (struct proc *) 0xfffff80071317000
>> error = 0
>> traced = 0
>> #27 0xffffffff814e2a8f in amd64_syscall (td=0xfffff800713124a0, traced=0) at /usr/src/sys/amd64/amd64/trap.c:952
>> sa = {code = 348, callp = 0xffffffff81d026d0, args = {140737488350418, 4, 34376556544, 2, 140737488345344, 34376515680, 0, 34376896576},
>> narg = 3}
>> error = 0
>> ksi = {ksi_link = {tqe_next = 0x202, tqe_prev = 0x202}, ksi_info = {si_signo = 2011958080, si_errno = -506, si_code = -2125728811, si_pid = -1,
>> si_uid = 4294960432, si_status = 32767, si_addr = 0x202, si_value = {sival_int = 1392864896, sival_ptr = 0xfffffe0653056e80,
>> sigval_int = 1392864896, sigval_ptr = 0xfffffe0653056e80}, _reason = {_fault = {_trapno = 1392864896}, _timer = {_timerid = 1392864896,
>> _overrun = -506}, _mesgq = {_mqd = 1392864896}, _poll = {_band = -2171860586880}, __spare__ = {__spare1__ = -2171860586880, __spare2__ = {
>> ---Type <return> to continue, or q <return> to quit---
>> 1392864896, -506, 0, -2048, -723702400, -507, 1899046048}}}}, ksi_flags = 2011958160, ksi_sigq = 0xffffffff80e3028d}
>> #28 0xffffffff814b9ddb in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:396
>> No locals.
>> #29 0x000000080092a17a in ?? ()
>> No symbol table info available.
>> Previous frame inner to this frame (corrupt stack?)
>> Current language: auto; currently minimal
>> (kgdb)q
>>
>> 16.02.2016, 12:10, "Steven Hartland" <killing at multiplay.co.uk>:
>>> I need the values of the vars specified, so you'll need to:
>>> print bp
>>> If it reports just an address try:
>>> print *bp
>>> etc.
>>>
>>> On 16/02/2016 08:08, DemIS wrote:
>>>> frame 8
>>>> #8 0xffffffff81c3669c in dbuf_read (db=0xfffff8013ca46380, zio=0x0, flags=6)
>>>> at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:573
>>>> 573 (void) arc_read(zio, db->db_objset->os_spa, db->db_blkptr,
>>>> Current language: auto; currently minimal
>>>>
>>>> frame 6
>>>> #6 0xffffffff81c2b9f8 in arc_get_data_buf (buf=<value optimized out>)
>>>> at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:2898
>>>> 2898 buf->b_data = zio_buf_alloc(size);
>>>>
>>>> 16.02.2016, 04:15, "Steven Hartland" <killing at multiplay.co.uk>:
>>>>> You don't need the system live just the kernel and the crash dump to get
>>>>> those values.
>>>>>
>>>>> On 16/02/2016 00:46, DemIS wrote:
>>>>>> Today I fell back to version 10.2 . Users work on the server . To reinstall 10.3 I need 4:00 .
>>>>>> Therefore, it is possible only on weekends . But this has the same effect on the version 10.2
>>>>>>
>>>>>> 16.02.2016, 02:55, "Steven Hartland" <killing at multiplay.co.uk>:
>>>>>>> That sounds like you have some pool data corruption, from the 10.3
>>>>>>> version dump can you print out the following:
>>>>>>> 1. frame 8: bp and size
>>>>>>> 2. frame 6: buf->b_hdr
>>>>>>>
>>>>>>> On 15/02/2016 23:26, DemIS wrote:
>>>>>>>> Any one knows about problem?
>>>>>>>> Server: SuperMicro Model:SYS-6026T-6RF+, MB:X8DTU-6F+, RAM 24Гб DDR3, two XEON
>>>>>>>> RAM: KVR1333D3E9S/4G - DDR3, 1333MHz, ECC, CL9, X8, 1.5V, Unbuffered, DIMM
>>>>>>>> Version:uname -a
>>>>>>>>
>>>>>>>> FreeBSD teo.some.loc 10.2-RELEASE-p12 FreeBSD 10.2-RELEASE-p12 #0: Sat Feb 13 18:04:04 MSK 2016 demis at teo.some.loc:/usr/obj/usr/src/sys/TEO amd64
>>>>>>>> (on GENERIC or custom kernel config persist too !!!)
>>>>>>>>
>>>>>>>> Memtest86+ v.4.40 (ECC mode) test - OK.
>>>>>>>> Every disk checked too (physically - mhdd, logically - zpool scrub, and additional checkit in external company recovery disk). No errors.
>>>>>>>>
>>>>>>>> Part of df -H
>>>>>>>> Filesystem Size Used Avail Capacity Mounted on
>>>>>>>> hdd/usr/wf 6,6T 4,1T 2,5T 62% /hdd/usr/wf
>>>>>>>>
>>>>>>>> zpool status hdd
>>>>>>>> pool: hdd
>>>>>>>> state: ONLINE
>>>>>>>> status: Some supported features are not enabled on the pool. The pool can
>>>>>>>> still be used, but some features are unavailable.
>>>>>>>> action: Enable all features using 'zpool upgrade'. Once this is done,
>>>>>>>> the pool may no longer be accessible by software that does not support
>>>>>>>> the features. See zpool-features(7) for details.
>>>>>>>> scan: scrub repaired 0 in 14h57m with 0 errors on Thu Feb 11 03:35:43 2016
>>>>>>>> config:
>>>>>>>>
>>>>>>>> NAME STATE READ WRITE CKSUM
>>>>>>>> hdd ONLINE 0 0 0
>>>>>>>> raidz2-0 ONLINE 0 0 0
>>>>>>>> mfid1p1 ONLINE 0 0 0
>>>>>>>> mfid2p1 ONLINE 0 0 0
>>>>>>>> mfid3p1 ONLINE 0 0 0
>>>>>>>> mfid4p1 ONLINE 0 0 0
>>>>>>>> mfid5p1 ONLINE 0 0 0
>>>>>>>>
>>>>>>>> errors: No known data errors
>>>>>>>>
>>>>>>>> hdd - is My zfs volume.
>>>>>>>> When I run command like:
>>>>>>>> rm /hdd/usr/some/path/to/file
>>>>>>>> or
>>>>>>>> rm /hdd/usr/some/path/to/folder
>>>>>>>> or
>>>>>>>> chown root:wheel /hdd/usr/some/path/to/file
>>>>>>>> or
>>>>>>>> chown root:wheel /hdd/usr/some/path/to/folder
>>>>>>>> or
>>>>>>>> setfacl ... to /hdd/usr/some/path/to/file
>>>>>>>>
>>>>>>>> I'm get kernel panic:
>>>>>>>> GNU gdb 6.1.1 [FreeBSD]
>>>>>>>> Copyright 2004 Free Software Foundation, Inc.
>>>>>>>> GDB is free software, covered by the GNU General Public License, and you are
>>>>>>>> welcome to change it and/or distribute copies of it under certain conditions.
>>>>>>>> Type "show copying" to see the conditions.
>>>>>>>> There is absolutely no warranty for GDB. Type "show warranty" for details.
>>>>>>>> This GDB was configured as "amd64-marcel-freebsd"...
>>>>>>>>
>>>>>>>> Unread portion of the kernel message buffer:
>>>>>>>> panic: solaris assert: c < (1ULL << 24) >> 9 (0x7fffffffffffff < 0x8000), file: /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c, line: 270
>>>>>>>> cpuid = 9
>>>>>>>> KDB: stack backtrace:
>>>>>>>> #0 0xffffffff80984ef0 at kdb_backtrace+0x60
>>>>>>>> #1 0xffffffff80948aa6 at vpanic+0x126
>>>>>>>> #2 0xffffffff80948973 at panic+0x43
>>>>>>>> #3 0xffffffff81c0222f at assfail3+0x2f
>>>>>>>> #4 0xffffffff81aa9d40 at zio_buf_alloc+0x50
>>>>>>>> #5 0xffffffff81a2b9f8 at arc_get_data_buf+0x358
>>>>>>>> #6 0xffffffff81a2e20a at arc_read+0x1ea
>>>>>>>> #7 0xffffffff81a3669c at dbuf_read+0x6ac
>>>>>>>> #8 0xffffffff81a3d8bf at dmu_spill_hold_existing+0xbf
>>>>>>>> #9 0xffffffff81a70dd7 at sa_attr_op+0x167
>>>>>>>> #10 0xffffffff81a72ffb at sa_lookup+0x4b
>>>>>>>> #11 0xffffffff81abc82a at zfs_rmnode+0x2ba
>>>>>>>> #12 0xffffffff81ada58e at zfs_freebsd_reclaim+0x4e
>>>>>>>> #13 0xffffffff80e73537 at VOP_RECLAIM_APV+0xa7
>>>>>>>> #14 0xffffffff809ec5b4 at vgonel+0x1b4
>>>>>>>> #15 0xffffffff809eca49 at vrecycle+0x59
>>>>>>>> #16 0xffffffff81ada52d at zfs_freebsd_inactive+0xd
>>>>>>>> #17 0xffffffff80e73427 at VOP_INACTIVE_APV+0xa7
>>>>>>>> Uptime: 9m31s
>>>>>>>> Dumping 1286 out of 24543 MB:..2%..12%..22%..32%..42%..51%..61%..71% (CTRL-C to abort) ..81%..91%
>>>>>>>>
>>>>>>>> Reading symbols from /boot/kernel/zfs.ko.symbols...done.
>>>>>>>> Loaded symbols for /boot/kernel/zfs.ko.symbols
>>>>>>>> Reading symbols from /boot/kernel/opensolaris.ko.symbols...done.
>>>>>>>> Loaded symbols for /boot/kernel/opensolaris.ko.symbols
>>>>>>>> Reading symbols from /boot/kernel/if_lagg.ko.symbols...done.
>>>>>>>> Loaded symbols for /boot/kernel/if_lagg.ko.symbols
>>>>>>>> Reading symbols from /boot/kernel/ums.ko.symbols...done.
>>>>>>>> Loaded symbols for /boot/kernel/ums.ko.symbols
>>>>>>>> Reading symbols from /boot/kernel/ipfw.ko.symbols...done.
>>>>>>>> Loaded symbols for /boot/kernel/ipfw.ko.symbols
>>>>>>>> #0 doadump (textdump=<value optimized out>) at pcpu.h:219
>>>>>>>> 219 pcpu.h: No such file or directory.
>>>>>>>> in pcpu.h
>>>>>>>> (kgdb) bt
>>>>>>>> #0 doadump (textdump=<value optimized out>) at pcpu.h:219
>>>>>>>> #1 0xffffffff80948702 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:451
>>>>>>>> #2 0xffffffff80948ae5 in vpanic (fmt=<value optimized out>, ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:758
>>>>>>>> #3 0xffffffff80948973 in panic (fmt=0x0) at /usr/src/sys/kern/kern_shutdown.c:687
>>>>>>>> #4 0xffffffff81c0222f in assfail3 (a=<value optimized out>, lv=<value optimized out>, op=<value optimized out>, rv=<value optimized out>,
>>>>>>>> f=<value optimized out>, l=<value optimized out>) at /usr/src/sys/modules/opensolaris/../../cddl/compat/opensolaris/kern/opensolaris_cmn_err.c:91
>>>>>>>> #5 0xffffffff81aa9d40 in zio_buf_alloc (size=0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:270
>>>>>>>> #6 0xffffffff81a2b9f8 in arc_get_data_buf (buf=<value optimized out>)
>>>>>>>> at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:2898
>>>>>>>> #7 0xffffffff81a2e20a in arc_read (pio=0xfffff80011791730, spa=0xfffff80011579000, bp=0xfffffe000aee7980, done=0xffffffff81a3a2d0 <dbuf_read_done>,
>>>>>>>> private=0xfffff8002244b000, priority=ZIO_PRIORITY_SYNC_READ, zio_flags=-528866606, arc_flags=0xfffffe06727fb3c4, zb=0xffffffff81a3a2d0)
>>>>>>>> at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:1551
>>>>>>>> #8 0xffffffff81a3669c in dbuf_read (db=0xfffff8002244b000, zio=0x0, flags=6)
>>>>>>>> at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:573
>>>>>>>> #9 0xffffffff81a3d8bf in dmu_spill_hold_existing (bonus=0xfffff800223bed20, tag=0x0, dbp=0xfffff800919966b8)
>>>>>>>> at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu.c:333
>>>>>>>> #10 0xffffffff81a70dd7 in sa_attr_op (hdl=0xfffff80091996690, bulk=0xfffffe06727fb528, count=1, data_op=SA_LOOKUP, tx=0x0)
>>>>>>>> at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/sa.c:310
>>>>>>>> #11 0xffffffff81a72ffb in sa_lookup (hdl=0xfffff80091996690, attr=<value optimized out>, buf=<value optimized out>, buflen=<value optimized out>)
>>>>>>>> at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/sa.c:1441
>>>>>>>> #12 0xffffffff81abc82a in zfs_rmnode (zp=0xfffff80091993730) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_dir.c:633
>>>>>>>> #13 0xffffffff81ada58e in zfs_freebsd_reclaim (ap=<value optimized out>)
>>>>>>>> at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:6569
>>>>>>>> #14 0xffffffff80e73537 in VOP_RECLAIM_APV (vop=<value optimized out>, a=<value optimized out>) at vnode_if.c:2019
>>>>>>>> #15 0xffffffff809ec5b4 in vgonel (vp=0xfffff800111733b0) at vnode_if.h:830
>>>>>>>> #16 0xffffffff809eca49 in vrecycle (vp=0xfffff800111733b0) at /usr/src/sys/kern/vfs_subr.c:2703
>>>>>>>> #17 0xffffffff81ada52d in zfs_freebsd_inactive (ap=<value optimized out>)
>>>>>>>> at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:6540
>>>>>>>> #18 0xffffffff80e73427 in VOP_INACTIVE_APV (vop=<value optimized out>, a=<value optimized out>) at vnode_if.c:1953
>>>>>>>> #19 0xffffffff809eb382 in vinactive (vp=0xfffff800111733b0, td=0xfffff800113d8000) at vnode_if.h:807
>>>>>>>> #20 0xffffffff809eb772 in vputx (vp=0xfffff800111733b0, func=2) at /usr/src/sys/kern/vfs_subr.c:2306
>>>>>>>> #21 0xffffffff809f401e in kern_rmdirat (td=<value optimized out>, fd=<value optimized out>, path=<value optimized out>, pathseg=<value optimized out>)
>>>>>>>> at /usr/src/sys/kern/vfs_syscalls.c:3842
>>>>>>>> #22 0xffffffff80d4b3e7 in amd64_syscall (td=0xfffff800113d8000, traced=0) at subr_syscall.c:134
>>>>>>>> #23 0xffffffff80d30acb in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:396
>>>>>>>> #24 0x00000008008914ea in ?? ()
>>>>>>>> Previous frame inner to this frame (corrupt stack?)
>>>>>>>> Current language: auto; currently minimal
>>>>>>>>
>>>>>>>> If setup FreeBSD 10.3 BETA (on GENERIC or custom kernel config):
>>>>>>>> Copyright 2004 Free Software Foundation, Inc.
>>>>>>>> GDB is free software, covered by the GNU General Public License, and you are
>>>>>>>> welcome to change it and/or distribute copies of it under certain conditions.
>>>>>>>> Type "show copying" to see the conditions.
>>>>>>>> There is absolutely no warranty for GDB. Type "show warranty" for details.
>>>>>>>> This GDB was configured as "amd64-marcel-freebsd"...
>>>>>>>>
>>>>>>>> Unread portion of the kernel message buffer:
>>>>>>>> panic: solaris assert: c < (1ULL << 24) >> 9 (0x7fffffffffffff < 0x8000), file: /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c, line: 273
>>>>>>>> cpuid = 13
>>>>>>>> KDB: stack backtrace:
>>>>>>>> #0 0xffffffff8098f000 at kdb_backtrace+0x60
>>>>>>>> #1 0xffffffff80951d06 at vpanic+0x126
>>>>>>>> #2 0xffffffff80951bd3 at panic+0x43
>>>>>>>> #3 0xffffffff81e0022f at assfail3+0x2f
>>>>>>>> #4 0xffffffff81cacc70 at zio_buf_alloc+0x50
>>>>>>>> #5 0xffffffff81c2b8f2 at arc_get_data_buf+0x262
>>>>>>>> #6 0xffffffff81c2b657 at arc_buf_alloc+0xc7
>>>>>>>> #7 0xffffffff81c2d601 at arc_read+0x1c1
>>>>>>>> #8 0xffffffff81c36ce9 at dbuf_read+0x6b9
>>>>>>>> #9 0xffffffff81c3e415 at dmu_spill_hold_existing+0xc5
>>>>>>>> #10 0xffffffff81c73707 at sa_attr_op+0x167
>>>>>>>> #11 0xffffffff81c75972 at sa_lookup+0x52
>>>>>>>> #12 0xffffffff81cbf8da at zfs_rmnode+0x2ba
>>>>>>>> #13 0xffffffff81cdd75e at zfs_freebsd_reclaim+0x4e
>>>>>>>> #14 0xffffffff80e81c27 at VOP_RECLAIM_APV+0xa7
>>>>>>>> #15 0xffffffff809f9581 at vgonel+0x221
>>>>>>>> #16 0xffffffff809f9a19 at vrecycle+0x59
>>>>>>>> #17 0xffffffff81cdd6fd at zfs_freebsd_inactive+0xd
>>>>>>>> Uptime: 11m11s
>>>>>>>> Dumping 1368 out of 24542 MB:..2%..11%..22%..31%..41%..51%..61%..71%..81%..91%
>>>>>>>>
>>>>>>>> Reading symbols from /boot/kernel/if_lagg.ko.symbols...done.
>>>>>>>> Loaded symbols for /boot/kernel/if_lagg.ko.symbols
>>>>>>>> Reading symbols from /boot/kernel/aio.ko.symbols...done.
>>>>>>>> Loaded symbols for /boot/kernel/aio.ko.symbols
>>>>>>>> Reading symbols from /boot/kernel/ichsmb.ko.symbols...done.
>>>>>>>> Loaded symbols for /boot/kernel/ichsmb.ko.symbols
>>>>>>>> Reading symbols from /boot/kernel/smbus.ko.symbols...done.
>>>>>>>> Loaded symbols for /boot/kernel/smbus.ko.symbols
>>>>>>>> Reading symbols from /boot/kernel/ipmi.ko.symbols...done.
>>>>>>>> Loaded symbols for /boot/kernel/ipmi.ko.symbols
>>>>>>>> Reading symbols from /boot/kernel/zfs.ko.symbols...done.
>>>>>>>> Loaded symbols for /boot/kernel/zfs.ko.symbols
>>>>>>>> Reading symbols from /boot/kernel/opensolaris.ko.symbols...done.
>>>>>>>> Loaded symbols for /boot/kernel/opensolaris.ko.symbols
>>>>>>>> Reading symbols from /boot/kernel/ums.ko.symbols...done.
>>>>>>>> Loaded symbols for /boot/kernel/ums.ko.symbols
>>>>>>>> Reading symbols from /boot/kernel/ipfw.ko.symbols...done.
>>>>>>>> Loaded symbols for /boot/kernel/ipfw.ko.symbols
>>>>>>>> #0 doadump (textdump=<value optimized out>) at pcpu.h:219
>>>>>>>> 219 pcpu.h: No such file or directory.
>>>>>>>> in pcpu.h
>>>>>>>> (kgdb) backtrace
>>>>>>>> #0 doadump (textdump=<value optimized out>) at pcpu.h:219
>>>>>>>> #1 0xffffffff80951962 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:486
>>>>>>>> #2 0xffffffff80951d45 in vpanic (fmt=<value optimized out>, ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:889
>>>>>>>> #3 0xffffffff80951bd3 in panic (fmt=0x0) at /usr/src/sys/kern/kern_shutdown.c:818
>>>>>>>> #4 0xffffffff81e0022f in assfail3 (a=<value optimized out>, lv=<value optimized out>, op=<value optimized out>, rv=<value optimized out>, f=<value optimized out>,
>>>>>>>> l=<value optimized out>) at /usr/src/sys/modules/opensolaris/../../cddl/compat/opensolaris/kern/opensolaris_cmn_err.c:91
>>>>>>>> #5 0xffffffff81cacc70 in zio_buf_alloc (size=0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:273
>>>>>>>> #6 0xffffffff81c2b8f2 in arc_get_data_buf (buf=<value optimized out>) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:3880
>>>>>>>> #7 0xffffffff81c2b657 in arc_buf_alloc (spa=<value optimized out>, size=<value optimized out>, tag=0x0, type=<value optimized out>)
>>>>>>>> at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:2057
>>>>>>>> #8 0xffffffff81c2d601 in arc_read (pio=0xfffff8000fad03b0, spa=0xfffff8000f63d000, bp=0xfffffe000e509980, done=0xffffffff81c3aed0 <dbuf_read_done>, private=0xfffff8000fdd6360,
>>>>>>>> priority=ZIO_PRIORITY_SYNC_READ, zio_flags=-2117882160, arc_flags=0xfffffe02925483c4, zb=0xfffff8000fdd6360)
>>>>>>>> at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:4397
>>>>>>>> #9 0xffffffff81c36ce9 in dbuf_read (db=0xfffff8000fdd6360, zio=0x0, flags=<value optimized out>)
>>>>>>>> at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:682
>>>>>>>> #10 0xffffffff81c3e415 in dmu_spill_hold_existing (bonus=0xfffff8001f312438, tag=0x0, dbp=0xfffff80062d4e7d0)
>>>>>>>> at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu.c:333
>>>>>>>> #11 0xffffffff81c73707 in sa_attr_op (hdl=0xfffff80062d4e770, bulk=0xfffffe0292548528, count=1, data_op=SA_LOOKUP, tx=0x0)
>>>>>>>> at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/sa.c:305
>>>>>>>> #12 0xffffffff81c75972 in sa_lookup (hdl=0xfffff80062d4e770, attr=<value optimized out>, buf=<value optimized out>, buflen=<value optimized out>)
>>>>>>>> at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/sa.c:1443
>>>>>>>> #13 0xffffffff81cbf8da in zfs_rmnode (zp=0xfffff80062d4c8a0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_dir.c:633
>>>>>>>> #14 0xffffffff81cdd75e in zfs_freebsd_reclaim (ap=<value optimized out>) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:6619
>>>>>>>> #15 0xffffffff80e81c27 in VOP_RECLAIM_APV (vop=<value optimized out>, a=<value optimized out>) at vnode_if.c:2019
>>>>>>>> #16 0xffffffff809f9581 in vgonel (vp=0xfffff8000f1beb10) at vnode_if.h:830
>>>>>>>> #17 0xffffffff809f9a19 in vrecycle (vp=0xfffff8000f1beb10) at /usr/src/sys/kern/vfs_subr.c:2951
>>>>>>>> #18 0xffffffff81cdd6fd in zfs_freebsd_inactive (ap=<value optimized out>) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:6590
>>>>>>>> #19 0xffffffff80e81b17 in VOP_INACTIVE_APV (vop=<value optimized out>, a=<value optimized out>) at vnode_if.c:1953
>>>>>>>> #20 0xffffffff809f8322 in vinactive (vp=0xfffff8000f1beb10, td=0xfffff8000f9f34b0) at vnode_if.h:807
>>>>>>>> #21 0xffffffff809f8712 in vputx (vp=0xfffff8000f1beb10, func=2) at /usr/src/sys/kern/vfs_subr.c:2547
>>>>>>>> #22 0xffffffff80a0137e in kern_rmdirat (td=<value optimized out>, fd=<value optimized out>, path=<value optimized out>, pathseg=<value optimized out>)
>>>>>>>> at /usr/src/sys/kern/vfs_syscalls.c:3964
>>>>>>>> #23 0xffffffff80d574bf in amd64_syscall (td=0xfffff8000f9f34b0, traced=0) at subr_syscall.c:141
>>>>>>>> #24 0xffffffff80d3c72b in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:396
>>>>>>>> #25 0x000000080089458a in ?? ()
>>>>>>>> Previous frame inner to this frame (corrupt stack?)
>>>>>>>> Current language: auto; currently minimal
>>>>>>>>
>>>>>>>> Crash folder (or file) have strange rights:
>>>>>>>> d---------+ 3 anna domain users 3 10 дек 10:32 01-Projcts
>>>>>>>> d---------+ 2 anna domain users 2 8 фев 21:46 02-Text
>>>>>>>>
>>>>>>>> How correct kernel panic?
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> freebsd-fs at freebsd.org mailing list
>>>>>>>> https://lists.freebsd.org/mailman/listinfo/freebsd-fs
>>>>>>>> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"
>>>>>>> _______________________________________________
>>>>>>> freebsd-fs at freebsd.org mailing list
>>>>>>> https://lists.freebsd.org/mailman/listinfo/freebsd-fs
>>>>>>> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"
More information about the freebsd-fs
mailing list