kern/114216: aio when using dt panic: lock "aiomtx" 0xc473f1f0 already initialized

Kirk Russell kirk at ba23.org
Mon Jul 2 14:10:03 UTC 2007


>Number:         114216
>Category:       kern
>Synopsis:       aio when using dt panic: lock "aiomtx" 0xc473f1f0 already initialized
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon Jul 02 14:10:02 GMT 2007
>Closed-Date:
>Last-Modified:
>Originator:     Kirk Russell
>Release:        7.0-CURRENT-200706
>Organization:
http://www.ba23.org/
>Environment:
FreeBSD amd.ott.ba23.org 7.0-CURRENT-200706 FreeBSD 7.0-CURRENT-200706 #0: Sat J
un 30 18:48:33 EDT 2007     root at amd.ott.ba23.org:/usr/obj/usr/src/sys/MYKERNEL
 i386

>Description:
I get a panic with a 7.0-CURRENT-200706 when using dt/AIO.


amd# kgdb kernel.debug /var/crash/vmcore.5
[GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd".

Unread portion of the kernel message buffer:
panic: lock "aiomtx" 0xc473f1f0 already initialized
cpuid = 0
Uptime: 7m38s
Physical memory: 1139 MB
Dumping 51 MB: 36 20 4

#0  doadump () at pcpu.h:172
172             __asm __volatile("movl %%fs:0,%0" : "=r" (td));
(kgdb) bt
#0  doadump () at pcpu.h:172
#1  0xc0730a3e in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409
#2  0xc0730d43 in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:563
#3  0xc0756e24 in lock_init (lock=0x0, class=0xc0ab2ae4, name=0xc47046e7 "aiomtx", type=0x0, flags=131072)
    at /usr/src/sys/kern/subr_lock.c:201
#4  0xc07258c4 in mtx_init (m=0xc473f1f0, name=0xc47046e7 "aiomtx", type=0x0, opts=0) at /usr/src/sys/kern/kern_mutex.c:714
#5  0xc4701f2e in ?? ()
#6  0xc473f1f0 in ?? ()
#7  0xc47046e7 in ?? ()
#8  0x00000000 in ?? ()
#9  0x00000000 in ?? ()
#10 0x00000000 in ?? ()
#11 0x28202800 in ?? ()
#12 0x00000000 in ?? ()
#13 0xe6c49c64 in ?? ()
#14 0xc470237c in ?? ()
#15 0xc472a4b8 in ?? ()
#16 0x00000003 in ?? ()
#17 0x00000004 in ?? ()
#18 0x00000002 in ?? ()
#19 0xe6c49c28 in ?? ()
#20 0xe6c49c1e in ?? ()
#21 0x0000000a in ?? ()
#22 0x00009c24 in ?? ()
#23 0x02570000 in ?? ()
#24 0x98c0ee98 in ?? ()
#25 0x000001ca in ?? ()
#26 0xda7a0000 in ?? ()
#27 0x00006400 in ?? ()
#28 0xc472a4b8 in ?? ()
#29 0x000018d5 in ?? ()
#30 0x00000000 in ?? ()
#31 0xc0ab49c0 in th4 ()
#32 0xc0ab4980 in th3 ()
#33 0xe6c49c50 in ?? ()
#34 0xc09819fe in __udivdi3 (a=Unhandled dwarf expression opcode 0x93
) at /usr/src/sys/libkern/udivdi3.c:47
Previous frame inner to this frame (corrupt stack?)
(kgdb) frame 3
#3  0xc0756e24 in lock_init (lock=0x0, class=0xc0ab2ae4, name=0xc47046e7 "aiomtx", type=0x0, flags=131072)
    at /usr/src/sys/kern/subr_lock.c:201
201             KASSERT(!lock_initalized(lock), ("lock \"%s\" %p already initialized",
(kgdb) print lock
$1 = (struct lock_object *) 0x0


Copyright (c) 1992-2007 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
    The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 7.0-CURRENT-200706 #0: Sat Jun 30 18:48:33 EDT 2007
    root at amd.ott.ba23.org:/usr/obj/usr/src/sys/MYKERNEL
WARNING: WITNESS option enabled, expect reduced performance.
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: AMD Athlon(tm) Processor (1210.79-MHz 686-class CPU)
  Origin = "AuthenticAMD"  Id = 0x642  Stepping = 2
  Features=0x183f9ff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR>
  AMD Features=0xc0440800<SYSCALL,<b18>,MMX+,3DNow!+,3DNow!>
real memory  = 1207877632 (1151 MB)
avail memory = 1168920576 (1114 MB)
kbd1 at kbdmux0
ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413)
acpi0: <ASUS A7V-133> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
acpi0: reservation of 0, a0000 (3) failed
acpi0: reservation of 100000, 47f00000 (3) failed
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0xe408-0xe40b on acpi0
cpu0: <ACPI CPU> on acpi0
acpi_throttle0: <ACPI CPU Throttling> on cpu0
acpi_button0: <Power Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
agp0: <VIA 82C8363 (Apollo KT133x/KM133) host to PCI bridge> on hostb0
pcib1: <PCI-PCI bridge> at device 1.0 on pci0
pci1: <PCI bus> on pcib1
vgapci0: <VGA-compatible display> port 0xd800-0xd8ff mem 0xe0000000-0xe3ffffff,0xd6000000-0xd600ffff irq 11 at device 0.0 on pci1
vgapci1: <VGA-compatible display> mem 0xd8000000-0xdbffffff,0xd5800000-0xd580ffff at device 0.1 on pci1
isab0: <PCI-ISA bridge> at device 4.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <VIA 82C686B UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xb800-0xb80f at device 4.1 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci0
ata1: [ITHREAD]
uhci0: <VIA 83C572 USB controller> port 0xb000-0xb01f irq 7 at device 4.3 on pci0
uhci0: [GIANT-LOCKED]
uhci0: [ITHREAD]
usb0: <VIA 83C572 USB controller> on uhci0
usb0: USB revision 1.0
uhub0: <VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb0
uhub0: 2 ports with 2 removable, self powered
pci0: <bridge> at device 4.4 (no driver attached)
pci0: <multimedia, video> at device 9.0 (no driver attached)
pci0: <multimedia, audio> at device 10.0 (no driver attached)
pci0: <input device> at device 10.1 (no driver attached)
ahc0: <Adaptec 2940 Ultra SCSI adapter> port 0x8800-0x88ff mem 0xd5000000-0xd5000fff irq 10 at device 11.0 on pci0
ahc0: [ITHREAD]
aic7880: Ultra Wide Channel A, SCSI Id=7, 16/253 SCBs
fxp0: <Intel 82559 Pro/100 Ethernet> port 0x8400-0x843f mem 0xd4800000-0xd4800fff,0xd4000000-0xd40fffff irq 11 at device 12.0 on pci0
miibus0: <MII bus> on fxp0
inphy0: <i82555 10/100 media interface> PHY 1 on miibus0
inphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
fxp0: Ethernet address: 00:04:ac:d3:7e:2f
fxp0: [ITHREAD]
ahc1: <Adaptec 2944 Ultra SCSI adapter> port 0x8000-0x80ff mem 0xd3800000-0xd3800fff irq 7 at device 13.0 on pci0
ahc1: [ITHREAD]
aic7880: Ultra Wide Channel A, SCSI Id=7, 16/253 SCBs
atapci1: <Promise PDC20265 UDMA100 controller> port 0x7800-0x7807,0x7400-0x7403,0x7000-0x7007,0x6800-0x6803,0x6400-0x643f mem 0xd3000000-0xd301ffff irq 10 at device 17.0 on pci0
atapci1: [ITHREAD]
ata2: <ATA channel 0> on atapci1
ata2: [ITHREAD]
ata3: <ATA channel 1> on atapci1
ata3: [ITHREAD]
fdc0: <floppy drive controller> port 0x3f2-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: [FILTER]
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
sio0: type 16550A
sio0: [FILTER]
sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0
sio1: type 16550A
sio1: [FILTER]
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
pmtimer0 on isa0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xccfff,0xd0000-0xd47ff,0xd8000-0xd87ff pnpid ORM0000 on isa0
ppc0: parallel port not found.
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounter "TSC" frequency 1210794194 Hz quality 800
Timecounters tick every 1.000 msec
Waiting 5 seconds for SCSI devices to settle
ad0: 8063MB <Maxtor 90845D4 GAS54112> at ata0-master UDMA33
acd0: DVDR <PIONEER DVD-RW DVR-110D/1.17> at ata1-master UDMA66
acd1: DVDROM <MATSHITADVD-ROM SR-8585/1A28> at ata1-slave UDMA33
(probe0:ahc0:0:0:0): inquiry data fails comparison at DV1 step
(probe0:ahc0:0:0:0): inquiry data fails comparison at DV1 step
(probe0:ahc0:0:0:0): inquiry data fails comparison at DV1 step
da0 at ahc0 bus 0 target 0 lun 0
da0: <COMPAQPC WDE9100W 1.01> Fixed Direct Access SCSI-2 device 
da0: 6.600MB/s transfers (16bit)
da0: Command Queueing Enabled
da0: 8678MB (17773500 512 byte sectors: 255H 63S/T 1106C)
Trying to mount root from ufs:/dev/ad0s1a

>How-To-Repeat:
This script appears to work okay with 6.0-RELEASE:

node116# kldload aio
node116# logout
$ uname -a
FreeBSD node116.on.kr 6.0-RELEASE FreeBSD 6.0-RELEASE #0: Tue Dec 20 21:06:21 ES
T 2005     root at node116.on.kr:/usr/src/sys/i386/compile/NODE116  i386
$ /usr/bin/time ./bstg0003.sh
       26.38 real         7.99 user         0.89 sys
$ echo $?
0
 

But with 7.0-CURRENT-200706, I get a panic instead.  Sometimes, you have
to run the script more than once:

amd# kldload aio
amd# logout
$ uname -a
FreeBSD amd.ott.ba23.org 7.0-CURRENT-200706 FreeBSD 7.0-CURRENT-200706 #0: Sat J
un 30 18:48:33 EDT 2007     root at amd.ott.ba23.org:/usr/obj/usr/src/sys/MYKERNEL
 i386
$ /usr/bin/time ./bstg0003.sh
       78.25 real         7.89 user         2.15 sys
$ /usr/bin/time ./bstg0003.sh
[... panic ...]



#!/bin/sh
DT=${DT:-/usr/local/sbin/dt}
TMPDIR=${TMPDIR:-/tmp}
TEMPFILE=${TMPDIR}/bstg0003.$$.$RANDOM

#
# See aio(4) about kernel POSIX AIO support.
# I have to run "kldload aio" before this test.
#
# If dt gives similar error messages, then your kernel does not
# appear to have AIO support:
#    dt: Child process NNNN, exiting because of signal 12
#    Bad system call (core dumped)
#
### kldload aio

#
# IMHO, dt is in the same league as fsx for finding interesting
# OS bugs.
#
# There is a FreeBSD ports for dt:
# http://www.freebsd.org/cgi/url.cgi?ports/sysutils/dt/pkg-descr
#
# Run dt (similar command line args to dd):
#    64 processes will each create a 6m file
#    will delete the file when finished
#    output block size of 16k
#    avoid printing any runtime stats to stdout
#    using POSIX AIO with 16 queued IOs per process
#
rm -f $TEMPFILE
$DT of=afile bs=16k limit=6m \
    disable=stats enable=aio aios=16 procs=64 > $TEMPFILE 2>&1

#
# I do not check the return code of dt.
# If any text appears on stdout/stderr, then you need to investigate.
#
if test -s $TEMPFILE ; then
    # oops -- appears to be a FAIL
    cat $TEMPFILE
    rm -f $TEMPFILE
    exit 1
fi

# done -- appears to be a PASS
rm -f $TEMPFILE
exit 0

>Fix:


>Release-Note:
>Audit-Trail:
>Unformatted:


More information about the freebsd-bugs mailing list