kern/182661: [PANIC] 9.2-RELEASE vm_page_unwire: page n's wire count is zero

[John Marshall]

>Number:         182661
>Category:       kern
>Synopsis:       [PANIC] 9.2-RELEASE vm_page_unwire: page n's wire count is zero
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sat Oct 05 07:00:00 UTC 2013
>Closed-Date:
>Last-Modified:
>Originator:     John Marshall
>Release:        FreeBSD 9.2-RELEASE amd64
>Organization:
Riverwillow Pty Ltd
>Environment:

System: FreeBSD ozsrv04.riverwillow.net.au 9.2-RELEASE FreeBSD 9.2-RELEASE #1 r255982: Wed Oct  2 15:03:22 AEST 2013     root at ozsrv04.riverwillow.net.au:/usr/obj/kits/src/sys/OZSRV04  amd64

System: FreeBSD rwsrv08.gfn.riverwillow.net.au 9.2-STABLE FreeBSD 9.2-STABLE #0 r256029: Fri Oct 4 07:22:42 AEST 2013 root at rwsrv08.gfn.riverwillow.net.au:/usr/obj/usr/src/sys/RWSRV08 amd64

	
>Description:

Since 9.2-RC4 I have observed this panic many times on each of two
servers running FreeBSD/amd64.  The panic usually occurs during system
shutdown when either ntpd or watchdogd are stopping.  Sometimes the
panic will occur when stopping either of those daemons independently of
a system shutdown.  I also saw this panic - once only - while the sftp
server was active accepting transfer of large (GB) files over the LAN.

I have about 30 dumps (from two systems) that look like the following.
Entry points and offsets are always identical.  Backtraces matching the
following account for all but one of the panics which I have seen, and
all appear to be triggered by ntpd or watchdogd exiting either while the
system is running or, usually, during shutdown.  These 30+ panics were
observed on 9.2-RC4, 9.2-RELEASE and 9.2-STABLE.  I took one of the
systems back to a fresh releng/9.1 (9.1-RELEASE-p7) and was unable to
trigger the panic.

  panic: vm_page_unwire: page <n>'s wire count is zero
  cpuid = <n> <-- anything from 0-7
  KDB: stack backtrace:
  #0 <n> at kdb_backtrace+0x68
  #1 <n> at panic+0x21a
  #2 <n> at vm_page_unwire+0x102
  #3 <n> at vm_fault_unwire+0xd2
  #4 <n> at vm_map_delete+0x171
  #5 <n> at vm_map_remove+0x5f
  #6 <n> at vmspace_exit+0xc9
  #7 <n> at exit1+0x72d
  #8 <n> at sys_sys_exit+0xe
  #9 <n> at amd64_syscall+0x3bf
  #10 <n> at Xfast_syscall+0xf7

The following is the exception.  This was when a 9.2-RC4 system panicked
with sftp server active.
 
  panic: vm_page_unwire: page 0xfffffe0239ecfc48's wire count is zero
  cpuid = 5
  KDB: stack backtrace:
  #0 0xffffffff80490268 at kdb_backtrace+0x68
  #1 0xffffffff8045630a at panic+0x21a
  #2 0xffffffff8068bbc2 at vm_page_unwire+0x102
  #3 0xffffffff804d857e at vfs_vmio_release+0x10e
  #4 0xffffffff804dadc8 at getnewbuf+0x468
  #5 0xffffffff804dbb2f at getblk+0x5df
  #6 0xffffffff80632bb1 at ffs_balloc_ufs2+0x15c1
  #7 0xffffffff8065a8d6 at ffs_write+0x246
  #8 0xffffffff8070a18f at VOP_WRITE_APV+0x11f
  #9 0xffffffff80505e77 at vn_write+0x1f7
  #10 0xffffffff80503b51 at vn_io_fault+0xb1
  #11 0xffffffff804a387c at dofilewrite+0x9c
  #12 0xffffffff804a3b54 at kern_writev+0x54
  #13 0xffffffff804a3bf5 at sys_write+0x65
  #14 0xffffffff806afd6f at amd64_syscall+0x3bf
  #15 0xffffffff8069a817 at Xfast_syscall+0xf7

See this thread on the -stable mailing list for more information.
http://lists.freebsd.org/pipermail/freebsd-stable/2013-September/075317.html

Please contact me for access to core.txt files or for extracting
information from dumps.  I still have most of the dumps and some of the
corresponding kernel.symbols files.

>How-To-Repeat:

I have only seen this FreeBSD/amd64 machines with 8 logical CPU's.  I
have not seen this on i386 (but none of my i386 systems have more than 2
logical CPU's).

 - Start ntpd
 - Stop ntpd (panics sometimes)

 - Start ntpd
 - Shutdown the system
 - Often panics when ntpd exiting
   
   Stopping ntpd.
   Waiting for PIDS: 1430
   panic: vm_page_unwire: page 0xfffffe0239144f38's wire count is zero
   cpuid = 7
   KDB: stack backtrace:
   #0 0xffffffff80490268 at kdb_backtrace+0x68
   #1 0xffffffff8045630a at panic+0x21a
   #2 0xffffffff8068bbc2 at vm_page_unwire+0x102
   #3 0xffffffff80678702 at vm_fault_unwire+0xd2
   (etc.)

>Fix:

	


>Release-Note:
>Audit-Trail:
>Unformatted: