kern/129719: Panic during shutdown, tcp_ctloutput: inp == NULL

[Dan Nelson]

>Number:         129719
>Category:       kern
>Synopsis:       Panic during shutdown, tcp_ctloutput: inp == NULL
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Dec 17 18:30:02 UTC 2008
>Closed-Date:
>Last-Modified:
>Originator:     Dan Nelson
>Release:        FreeBSD 7.1-PRERELEASE i386
>Organization:
>Environment:

System: FreeBSD filesrv.allantgroup.com 7.1-PRERELEASE FreeBSD 7.1-PRERELEASE #19: Sat Nov 15 17:23:42 CST 2008 zsh at filesrv.allantgroup.com:/usr/src/sys/i386/compile/FILESRV i386

	
>Description:
	

I've been trying to solve an intermittent connectivity problem where a
server stops seeing incoming packets.  It happened today, and when the
system was shutting down, it paniced and rebooted.  The gdb stack trace
is a little mangled due to inlined functions, but the trap was in
tcp_usrreq.c, line 1266.  Looks like it was trying to reconnect a TCP
NFS mount.

1255    int 
1256    tcp_ctloutput(struct socket *so, struct sockopt *sopt)
1257    {   
1258            int     error, opt, optval;
1259            struct  inpcb *inp;
1260            struct  tcpcb *tp;
1261            struct  tcp_info ti;
1262
1263            error = 0;
1264            inp = sotoinpcb(so);
1265            KASSERT(inp != NULL, ("tcp_ctloutput: inp == NULL"));
1266 *          INP_WLOCK(inp);
1267            if (sopt->sopt_level != IPPROTO_TCP) {


I don't have INVARIANTS enabled, which would have triggered the KASSERT
one line up.  I've got the core dump if more info is needed.



Fatal trap 12: page fault while in kernel mode
cpuid = 3; apic id = 07
fault virtual address   = 0xac
fault code              = supervisor write, page not present
instruction pointer     = 0x20:0xc07f58fd
stack pointer           = 0x28:0xef7fb908
frame pointer           = 0x28:0xef7fba1c
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, def32 1, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 3221 (nfsiod 0)
trap number             = 12
panic: page fault
cpuid = 3
KDB: stack backtrace:
db_trace_self_wrapper(c098f9a8,ef7fb7a4,c06bd4a7,c09ac250,3,...) at db_trace_self_wrapper+0x26
kdb_backtrace(c09ac250,3,c095d6a2,ef7fb7b0,3,...) at kdb_backtrace+0x29
panic(c095d6a2,c09ad488,c5766794,1,1,...) at panic+0x114
trap_fatal(c0a5aec0,0,2,8,0,...) at trap_fatal+0x32e
trap_pfault(0,0,c1060018,c1060014,c,...) at trap_pfault+0x244
trap(ef7fb8c8) at trap+0x3af
calltrap() at calltrap+0x6
--- trap 0xc, eip = 0xc07f58fd, esp = 0xef7fb908, ebp = 0xef7fba1c ---
tcp_ctloutput(c71a0680,ef7fbac8,4,4,58,...) at tcp_ctloutput+0x21
sosetopt(c71a0680,ef7fbac8,58,c099dedd,1f4,...) at sosetopt+0x666
nfs_connect(c54e4d20,c6208000,0,c0a11384,100,...) at nfs_connect+0x6c4
nfs_reconnect(c6208000,c6208044,153,c099e71c,0,...) at nfs_reconnect+0x16f
nfs_request(c81e0228,c56f4000,7,0,c6f46b00,...) at nfs_request+0x831
nfs_writerpc(c81e0228,ef7fbc84,c6f46b00,ef7fbca8,ef7fbca4,...) at nfs_writerpc+0x2ea
nfs_doio(c81e0228,d8e7e074,c6f46b00,0,dbba0,...) at nfs_doio+0x616
nfssvc_iod(c0a6e220,ef7fbd38,0,0,80eb7a0,...) at nfssvc_iod+0x2b9
fork_exit(c084192c,c0a6e220,ef7fbd38) at fork_exit+0x99
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xef7fbd70, ebp = 0 ---
Uptime: 28d0h22m4s
Physical memory: 2039 MB
Dumping 280 MB: 265 249 233 217 201 185 169 153 137 121 105 89 73 57 41 25 9

Reading symbols from /boot/modules/cpu.ko...done.
Loaded symbols for /boot/modules/cpu.ko
Reading symbols from /boot/kernel/matrix_saver.ko...Reading symbols from /boot/kernel/matrix_saver.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/matrix_saver.ko
#0  doadump () at pcpu.h:196
196     pcpu.h: No such file or directory.
        in pcpu.h
(kgdb) (kgdb) Undefined command: "kldsyms".  Try "help".
(kgdb) #0  doadump () at pcpu.h:196
#1  0xc06bd1e6 in boot (howto=260) at ../../../kern/kern_shutdown.c:418
#2  0xc06bd4e3 in panic (fmt=Variable "fmt" is not available) at ../../../kern/kern_shutdown.c:574
#3  0xc091cb09 in trap_fatal (frame=0xef7fb8c8, eva=172) at ../../../i386/i386/trap.c:939
#4  0xc091cd59 in trap_pfault (frame=0xef7fb8c8, usermode=0, eva=172) at ../../../i386/i386/trap.c:852
#5  0xc091d6eb in trap (frame=0xef7fb8c8) at ../../../i386/i386/trap.c:530
#6  0xc0904a2b in calltrap () at ../../../i386/i386/exception.s:159
#7  0xc07f58fd in tcp_ctloutput (so=0xc71a0680, sopt=0xef7fbac8) at atomic.h:149
#8  0xc071024d in sosetopt (so=0xc71a0680, sopt=0xef7fbac8) at ../../../kern/uipc_socket.c:2339
#9  0xc083ba5c in nfs_connect (nmp=0xc54e4d20, rep=0xc6208000) at ../../../nfsclient/nfs_socket.c:428
#10 0xc083bf9a in nfs_reconnect (rep=0xc6208000) at ../../../nfsclient/nfs_socket.c:542
#11 0xc083e6e5 in nfs_request (vp=0xc81e0228, mrest=0xc56f4000, procnum=7, td=0x0, cred=0xc6f46b00, mrp=0xef7fbc08, mdp=0xef7fbc04, dposp=0xef7fbc0c) at ../../../nfsclient/nfs_socket.c:737
#12 0xc0847577 in nfs_writerpc (vp=0xc81e0228, uiop=0xef7fbc84, cred=0xc6f46b00, iomode=0xef7fbca8, must_commit=0xef7fbca4) at ../../../nfsclient/nfs_vnops.c:1183
#13 0xc083539f in nfs_doio (vp=0xc81e0228, bp=0xd8e7e074, cr=0xc6f46b00, td=0x0) at ../../../nfsclient/nfs_bio.c:1683
#14 0xc0841be5 in nfssvc_iod (instance=0xc0a6e220) at ../../../nfsclient/nfs_nfsiod.c:282
#15 0xc069a351 in fork_exit (callout=0xc084192c <nfssvc_iod>, arg=0xc0a6e220, frame=0xef7fbd38) at ../../../kern/kern_fork.c:804
#16 0xc0904aa0 in fork_trampoline () at ../../../i386/i386/exception.s:264
(kgdb) #                                   



>How-To-Repeat:
	
>Fix:

	


>Release-Note:
>Audit-Trail:
>Unformatted: