kern/162751: [zfs] [panic] kernel panics during file operations

Jacek Kalamarz jkalamarz at gmail.com
Tue Nov 22 10:40:12 UTC 2011


>Number:         162751
>Category:       kern
>Synopsis:       [zfs] [panic] kernel panics during file operations
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Tue Nov 22 10:40:06 UTC 2011
>Closed-Date:
>Last-Modified:
>Originator:     Jacek Kalamarz
>Release:        8.2-RELEASE-p4
>Organization:
>Environment:
FreeBSD kim.rolskiego.net 8.2-RELEASE-p4 FreeBSD 8.2-RELEASE-p4 #0: Tue Oct 25 10:15:05 CEST 2011     root at kim.rolskiego.net:/usr/obj/usr/src/sys/GENERIC  amd64

Celeron 1.2GHz, 2GB RAM, zpool created on 1TB partition

ZFS details:
simson at kim:usr/src/sys/GENERIC$ zpool status
  pool: tank
 state: ONLINE
 scrub: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        tank        ONLINE       0     0     0
          ad4s1d    ONLINE       0     0     0

errors: No known data errors
simson at kim:usr/src/sys/GENERIC$ zfs list
NAME           USED  AVAIL  REFER  MOUNTPOINT
tank           111G   803G    23K  none
tank/home     47.7G  52.3G  16.3G  /home
tank/storage  54.1G   246G  54.1G  /storage
tank/tmp      30.2M  50.0G  30.2M  /tmp
tank/usr      2.19G  2.81G  2.19G  /usr
tank/var      4.58G  5.42G  4.58G  /var
simson at kim:usr/src/sys/GENERIC$ zfs list -H -t snapshot | wc -l
      36

>Description:
Since ZFS is used on the machine, the machine crashes about once a week.
Previously (using only UFS2 partitions), the machine was stable for 3 months.

The logs show exactly the same code line each time:


simson at kim:usr/src/sys/GENERIC$ kgdb kernel.debug /var/crash/vmcore.0
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:


Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0x460700000c9e
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff80820f37
stack pointer           = 0x28:0xffffff8092280770
frame pointer           = 0x28:0xffffff8092280800
code segment            = base 0x0, limit 0xfffff, type 0x1b
                       = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 38 (arc_reclaim_thread)
trap number             = 12
panic: page fault
cpuid = 0
KDB: stack backtrace:
#0 0xffffffff805f4e0e at kdb_backtrace+0x5e
#1 0xffffffff805c2d07 at panic+0x187
#2 0xffffffff808ac630 at trap_fatal+0x291
#3 0xffffffff808aca0f at trap_pfault+0x28f
#4 0xffffffff808aceef at trap+0x3df
#5 0xffffffff80894fe4 at calltrap+0x8
#6 0xffffffff80821932 at vm_page_remove+0x32
#7 0xffffffff80821a7d at vm_page_free_toq+0x6d
#8 0xffffffff8082085b at vm_object_page_remove+0x11b
#9 0xffffffff80818c33 at vm_map_delete+0x313
#10 0xffffffff80818d41 at vm_map_remove+0x51
#11 0xffffffff8080d6a5 at uma_large_free+0x55
#12 0xffffffff805aff97 at free+0x77
#13 0xffffffff80e36351 at arc_buf_destroy+0x101
#14 0xffffffff80e39614 at arc_evict+0x2f4
#15 0xffffffff80e3a6ec at arc_adjust+0x1bc
#16 0xffffffff80e3a9b0 at arc_reclaim_thread+0x1a0
#17 0xffffffff805994f8 at fork_exit+0x118
Uptime: 11d9h8m14s
Physical memory: 1997 MB
Dumping 1632 MB: 1617 1601 1585 1569 1553 1537 1521 1505 1489 1473 1457 1441 1425 1409 1393 1377 1361 1345 1329 1313 1297 1281 1265 1249 1233 1217 1201 1185 1169 1153 1137 1121 1105 1089 1073 1057 1041 1025 1009 993 977 961 945 929 913 897 881 865 849 833 817 801 785 769 753 737 721 705 689 673 657 641 625 609 593 577 561 545 529 513 497 481 465 449 433 417 401 385 369 353 337 321 305 289 273 257 241 225 209 193 177 161 145 129 113 97 81 65 49 33 17 1

Reading symbols from /boot/kernel/zfs.ko...Reading symbols from /boot/kernel/zfs.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/zfs.ko
Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from /boot/kernel/opensolaris.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/opensolaris.ko
Reading symbols from /usr/local/modules/fuse.ko...done.
Loaded symbols for /usr/local/modules/fuse.ko
Reading symbols from /boot/kernel/snp.ko...Reading symbols from /boot/kernel/snp.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/snp.ko
#0  doadump () at pcpu.h:224
224             __asm("movq %%gs:0,%0" : "=r" (td));
(kgdb) bt
#0  doadump () at pcpu.h:224
#1  0xffffffff805c28be in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:419
#2  0xffffffff805c2cf1 in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:592
#3  0xffffffff808ac630 in trap_fatal (frame=0xc, eva=Variable "eva" is not available.
) at /usr/src/sys/amd64/amd64/trap.c:783
#4  0xffffffff808aca0f in trap_pfault (frame=0xffffff80922806c0, usermode=0) at /usr/src/sys/amd64/amd64/trap.c:699
#5  0xffffffff808aceef in trap (frame=0xffffff80922806c0) at /usr/src/sys/amd64/amd64/trap.c:449
#6  0xffffffff80894fe4 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:224
#7  0xffffffff80820f37 in vm_page_splay (pindex=223004, root=0x460700000c66) at /usr/src/sys/vm/vm_page.c:624
#8  0xffffffff80821932 in vm_page_remove (m=0xffffff007b206c70) at /usr/src/sys/vm/vm_page.c:741
#9  0xffffffff80821a7d in vm_page_free_toq (m=0xffffff007b206c70) at /usr/src/sys/vm/vm_page.c:1562
#10 0xffffffff8082085b in vm_object_page_remove (object=0xffffffff80b957a0, start=222976, end=223008, clean_only=0) at /usr/src/sys/vm/vm_object.c:1788
#11 0xffffffff80818c33 in vm_map_delete (map=0xffffff00010000e8, start=Variable "start" is not available.
) at /usr/src/sys/vm/vm_map.c:2715
#12 0xffffffff80818d41 in vm_map_remove (map=0xffffff00010000e8, start=18446743524867047424, end=18446743524867178496) at /usr/src/sys/vm/vm_map.c:2846
#13 0xffffffff8080d6a5 in uma_large_free (slab=0xffffff00291c2470) at /usr/src/sys/vm/uma_core.c:3084
#14 0xffffffff805aff97 in free (addr=0xffffff8036700000, mtp=0xffffffff80f277c0) at /usr/src/sys/kern/kern_malloc.c:506
#15 0xffffffff80e36351 in arc_buf_destroy (buf=0xffffff002c2710d8, recycle=Variable "recycle" is not available.
) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:1497
#16 0xffffffff80e39614 in arc_evict (state=0xffffffff80f11b00, spa=0, bytes=35238618, recycle=0, type=ARC_BUFC_DATA) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:1780
#17 0xffffffff80e3a6ec in arc_adjust () at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:1993
#18 0xffffffff80e3a9b0 in arc_reclaim_thread (dummy=Variable "dummy" is not available.
) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:2251
#19 0xffffffff805994f8 in fork_exit (callout=0xffffffff80e3a810 <arc_reclaim_thread>, arg=0x0, frame=0xffffff8092280c40) at /usr/src/sys/kern/kern_fork.c:845
#20 0xffffffff808954ae in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:565
#21 0x0000000000000000 in ?? ()
#22 0x0000000000000000 in ?? ()
#23 0x0000000000000001 in ?? ()
#24 0x0000000000000000 in ?? ()
#25 0x0000000000000000 in ?? ()
#26 0x0000000000000000 in ?? ()
#27 0x0000000000000000 in ?? ()
#28 0x0000000000000000 in ?? ()
#29 0x0000000000000000 in ?? ()
#30 0x0000000000000000 in ?? ()
#31 0x0000000000000000 in ?? ()
#32 0x0000000000000000 in ?? ()
#33 0x0000000000000000 in ?? ()
#34 0x0000000000000000 in ?? ()
#35 0x0000000000000000 in ?? ()
#36 0x0000000000000000 in ?? ()
#37 0x0000000000000000 in ?? ()
#38 0x0000000000000000 in ?? ()
#39 0x0000000000000000 in ?? ()
#40 0x0000000000000000 in ?? ()
#41 0x0000000000000000 in ?? ()
#42 0x0000000000000000 in ?? ()
#43 0x0000000000000000 in ?? ()
#44 0x0000000000000000 in ?? ()
#45 0xffffffff80b67d80 in affinity ()
#46 0x0000000000000000 in ?? ()
#47 0x0000000000000000 in ?? ()
#48 0xffffff00019648c0 in ?? ()
#49 0xffffff80922806a0 in ?? ()
#50 0xffffff8092280648 in ?? ()
#51 0xffffff00015c6000 in ?? ()
#52 0xffffffff805e81b9 in sched_switch (td=0xffffffff80e3a810, newtd=0x0, flags=Variable "flags" is not available.
) at /usr/src/sys/kern/sched_ule.c:1852
Previous frame inner to this frame (corrupt stack?)
(kgdb) f 7
#7  0xffffffff80820f37 in vm_page_splay (pindex=223004, root=0x460700000c66) at /usr/src/sys/vm/vm_page.c:624
624                             lefttreemax->right = root;
(kgdb) l *0xffffffff80820f37
0xffffffff80820f37 is in vm_page_splay (/usr/src/sys/vm/vm_page.c:598).
598                     if (pindex < root->pindex) {
(kgdb) l 595,630
595                     return (root);
596             lefttreemax = righttreemin = &dummy;
597             for (;; root = y) {
598                     if (pindex < root->pindex) {
599                             if ((y = root->left) == NULL)
600                                     break;
601                             if (pindex < y->pindex) {
602                                     /* Rotate right. */
603                                     root->left = y->right;
604                                     y->right = root;
605                                     root = y;
606                                     if ((y = root->left) == NULL)
607                                             break;
608                             }
609                             /* Link into the new root's right tree. */
610                             righttreemin->left = root;
611                             righttreemin = root;
612                     } else if (pindex > root->pindex) {
613                             if ((y = root->right) == NULL)
614                                     break;
615                             if (pindex > y->pindex) {
616                                     /* Rotate left. */
617                                     root->right = y->left;
618                                     y->left = root;
619                                     root = y;
620                                     if ((y = root->right) == NULL)
621                                             break;
622                             }
623                             /* Link into the new root's left tree. */
624                             lefttreemax->right = root;
625                             lefttreemax = root;
626                     } else
627                             break;
628             }
629             /* Assemble the new root. */
630             lefttreemax->right = root->left;
(kgdb) p righttreemin
$7 = 0xffffff8092280780
(kgdb) p lefttreemax
$8 = 0xffffff8092280780
(kgdb) p &dummy
$9 = (struct vm_page *) 0xffffff8092280780
(kgdb) p dummy
$10 = {pageq = {tqe_next = 0x0, tqe_prev = 0xffffff007ce2f880}, listq = {tqe_next = 0xffffff8092280850, tqe_prev = 0xffffffff8080ff7b}, left = 0x0, right = 0xffffff007cbb5158, object = 0xffffff007ce2f800,
 pindex = 18446742974224550080, phys_addr = 18446742976246783680, md = {pv_list = {tqh_first = 0xffffff007af92548, tqh_last = 0xffffff00019648c0}, pat_mode = 513}, queue = 64 '@', segind = 8 '\b',
 flags = 37416, order = 128 '\200', pool = 255 'y', cow = 65535, wire_count = 2156563222, hold_count = -1, oflags = 65535, act_count = 192 'A', busy = 72 'H', valid = 150 '\226', dirty = 1 '\001'}
(kgdb) p y
$11 = 0xffffff007b206c70
(kgdb) p *y
$12 = {pageq = {tqe_next = 0x0, tqe_prev = 0xffffffff80b95ec0}, listq = {tqe_next = 0xffffff00794f2b48, tqe_prev = 0xffffff007b8530c0}, left = 0x0, right = 0xffffff007bead320, object = 0xffffffff80b955a0,
 pindex = 223004, phys_addr = 1113141248, md = {pv_list = {tqh_first = 0x0, tqh_last = 0xffffff007b206cb8}, pat_mode = 6}, queue = 0 '\0', segind = 2 '\002', flags = 2048, order = 13 '\r', pool = 0 '\0',
 cow = 0, wire_count = 0, hold_count = 0, oflags = 0, act_count = 0 '\0', busy = 0 '\0', valid = 255 'y', dirty = 0 '\0'}
(kgdb) p pindex
$13 = 223004
(kgdb) p root
$14 = 0x460700000c66
(kgdb) p *root
Cannot access memory at address 0x460700000c66
(kgdb)


Other dumps:
vmcore.1:
(kgdb) p righttreemin
$1 = 0xffffff8092280780
(kgdb) p lefttreemax
$2 = 0xffffff8092280780
(kgdb) p y->pindex
$3 = 128084
(kgdb) p pindex
$4 = 128084
(kgdb) p root
$5 = 0x592e00000c66

vmcore.3:
(kgdb) p righttreemin
$1 = 0xffffff8092248780
(kgdb) p lefttreemax
$2 = 0xffffff8092248780
(kgdb) p pindex
$3 = 121576
(kgdb) p y->pindex
$4 = 121576
(kgdb) p root
$5 = 0x2b1600000c66



>How-To-Repeat:
Probably crashes after large load (tar, http serving, etc.)
>Fix:


>Release-Note:
>Audit-Trail:
>Unformatted:


More information about the freebsd-bugs mailing list