kern/133439: Kernel Panic in kern_vfs

Mon Apr 6 13:40:05 PDT 2009

>Number:         133439
>Category:       kern
>Synopsis:       Kernel Panic in kern_vfs
>Confidential:   no
>Severity:       serious
>Priority:       low
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon Apr 06 20:40:03 UTC 2009
>Closed-Date:
>Last-Modified:
>Originator:     John Hickey
>Release:        RELENG_6_4
>Organization:
>Environment:
FreeBSD users.isi.deterlab.net 6.4-RELEASE-p3 FreeBSD 6.4-RELEASE-p3 #5: Mon Apr  6 12:35:02 PDT 2009     unknown@:/usr/obj/usr/src/sys/USERS  i386

>Description:
Unread portion of the kernel message buffer:
kernel trap 12 with interrupts disabled

Fatal trap 12: page fault while in kernel mode
cpuid = 7; apic id = 07
fault virtual address	= 0x104
fault code		= supervisor read, page not present
instruction pointer	= 0x20:0xc05ba1e1
stack pointer	        = 0x28:0xeb571a74
frame pointer	        = 0x28:0xeb571a80
code segment		= base 0x0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, def32 1, gran 1
processor eflags	= resume, IOPL = 0
current process		= 826 (nfsd)
trap number		= 12
panic: page fault
cpuid = 7
Uptime: 23h44m52s
Dumping 3578 MB (2 chunks)
  chunk 0: 1MB (159 pages) ... ok
  chunk 1: 3579MB (916048 pages) 3563 3547 3531 3515 3499 3483 3467 3451 3435 3419 3403 3387 3371 3355 3339 3323 3307 3291 3275 3259 3243 3227 3211 3195 3179 3163 3147 3131 3115 3099 3083 3067 3051 3035 3019 3003 2987 2971 2955 2939 2923 2907 2891 2875 2859 2843 2827 2811 2795 2779 2763 2747 2731 2715 2699 2683 2667 2651 2635 2619 2603 2587 2571 2555 2539 2523 2507 2491 2475 2459 2443 2427 2411 2395 2379 2363 2347 2331 2315 2299 2283 2267 2251 2235 2219 2203 2187 2171 2155 2139 2123 2107 2091 2075 2059 2043 2027 2011 1995 1979 1963 1947 1931 1915 1899 1883 1867 1851 1835 1819 1803 1787 1771 1755 1739 1723 1707 1691 1675 1659 1643 1627 1611 1595 1579 1563 1547 1531 1515 1499 1483 1467 1451 1435 1419 1403 1387 1371 1355 1339 1323 1307 1291 1275 1259 1243 1227 1211 1195 1179 1163 1147 1131 1115 1099 1083 1067 1051 1035 1019 1003 987 971 955 939 923 907 891 875 859 843 827 811 795 779 763 747 731 715 699 683 667 651 635 619 603 587 571 555 539 523 507 491 475 459 443 427 411 395 
 379 363 347 331 315 299 283 267 251 235 219 203 187 171 155 139 123 107 91 75 59 43 27 11

#0  doadump () at pcpu.h:165
165	pcpu.h: No such file or directory.
	in pcpu.h(kgdb) bt
#0  doadump () at pcpu.h:165
#1  0xc05c4ce6 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:410
#2  0xc05c503d in panic (fmt=0xc0780dc4 "%s") at /usr/src/sys/kern/kern_shutdown.c:566
#3  0xc0759ea6 in trap_fatal (frame=0xeb571a34, eva=260) at /usr/src/sys/i386/i386/trap.c:838
#4  0xc07595be in trap (frame=
      {tf_fs = -928776184, tf_es = -346619864, tf_ds = -1067253720, tf_edi = -921424896, tf_esi = 4, tf_ebp = -346613120, tf_isp = -346613152, tf_ebx = -921123716, tf_edx = 6, tf_ecx = 4, tf_eax = 1, tf_trapno = 12, tf_err = 0, tf_eip = -1067736607, tf_cs = 32, tf_eflags = 65538, tf_esp = -921123840, tf_ss = 0}) at /usr/src/sys/i386/i386/trap.c:270
#5  0xc07446aa in calltrap () at /usr/src/sys/i386/i386/exception.s:139
#6  0xc05ba1e1 in _mtx_lock_sleep (m=0xc918c47c, tid=3373542400, opts=0, file=0x0, line=0) at /usr/src/sys/kern/kern_mutex.c:546
#7  0xc0618954 in vfs_export_lookup (mp=0xc909e7c8, nam=0x6) at /usr/src/sys/kern/vfs_export.c:380
#8  0xc06189d2 in vfs_stdcheckexp (mp=0xc909e7c8, nam=0xccf57360, extflagsp=0x1, credanonp=0x1) at /usr/src/sys/kern/vfs_export.c:415
#9  0xc06b5075 in nfsrv_fhtovp (fhp=0xeb571b68, lockflag=1, vpp=0xeb571b30, cred=0xca191500, slp=0xca67ae80, nam=0xccf57360, 
    rdonlyp=0x1, pubflag=1) at /usr/src/sys/nfsserver/nfs_srvsubs.c:1097
#10 0xc06a3629 in nfsrv_getattr (nfsd=0xca1ebd00, slp=0xca67ae80, td=0x1, mrq=0xeb571c98) at /usr/src/sys/nfsserver/nfs_serv.c:268
#11 0xc06b62b1 in nfssvc_nfsd (td=0x1) at /usr/src/sys/nfsserver/nfs_syscalls.c:474
#12 0xc06b5a90 in nfssvc (td=0xc9142c00, uap=0xeb571d04) at /usr/src/sys/nfsserver/nfs_syscalls.c:181
#13 0xc075a1eb in syscall (frame=
      {tf_fs = 59, tf_es = 59, tf_ds = 59, tf_edi = 1, tf_esi = 0, tf_ebp = -1077941464, tf_isp = -346612380, tf_ebx = 64, tf_edx = 1746200344, tf_ecx = 26, tf_eax = 155, tf_trapno = 12, tf_err = 2, tf_eip = 1745609635, tf_cs = 51, tf_eflags = 662, tf_esp = -1077941492, tf_ss = 59}) at /usr/src/sys/i386/i386/trap.c:984
#14 0xc07446ff in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:200
#15 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)

>How-To-Repeat:
This system does a good bit of NFS serving and the exports map is modified on a regular basis (.  The system is an 8 way machine.

I'm no expert at what is going on here, but I suspect that we are getting by doing a vfs_export_lookup while vfs_free_addrlist is going.

(kgdb) up 7
#7  0xc0618954 in vfs_export_lookup (mp=0xc909e7c8, nam=0x6) at /usr/src/sys/kern/vfs_export.c:380
380					RADIX_NODE_HEAD_LOCK(rnh);
(kgdb) list
375			 */
376			if (nam != NULL) {
377				saddr = nam;
378				rnh = nep->ne_rtable[saddr->sa_family];
379				if (rnh != NULL) {
380					RADIX_NODE_HEAD_LOCK(rnh);
381					np = (struct netcred *)
382					    (*rnh->rnh_matchaddr)(saddr, rnh);
383					RADIX_NODE_HEAD_UNLOCK(rnh);
384					if (np && np->netc_rnodes->rn_flags & RNF_ROOT)

There is a comment on line 217 of vfs_free_addrlist noting that it is not SMP safe.  So between lines 379 and 380 rnh is being freed by vfs_free_addrlist, but we still try to call RADIX_NODE_HEAD_LOCK on the old address and get a page fault.   It appears when changing a mountpoint the address hash is deleted completely and then recreated.
>Fix:

>Release-Note:
>Audit-Trail:
>Unformatted: