Questions about witness reports on sparc64

Fri Mar 5 01:47:11 UTC 2010

Folks,

I'm pretty new here, so feel free to use the clue stick :)

Playing with current- on Sun Fire V210 I see more or less
consistent lock reversal reports like the following:

lock order reversal:
 1st 0xc1c85fb8 bufwait (bufwait) @ /usr/src/sys/kern/vfs_bio.c:2559
 2nd 0xfffff80002784600 dirhash (dirhash) @
/usr/src/sys/ufs/ufs/ufs_dirhash.c:285
KDB: stack backtrace:
_witness_debugger() at _witness_debugger+0x38
witness_checkorder() at witness_checkorder+0xcf8
_sx_xlock() at _sx_xlock+0x9c
ufsdirhash_acquire() at ufsdirhash_acquire+0x30
ufsdirhash_add() at ufsdirhash_add+0x4
ufs_direnter() at ufs_direnter+0x75c
ufs_makeinode() at ufs_makeinode+0x4dc
ufs_create() at ufs_create+0x40
VOP_CREATE_APV() at VOP_CREATE_APV+0x108
vn_open_cred() at vn_open_cred+0x1fc
vn_open() at vn_open+0x1c
kern_openat() at kern_openat+0x10c
kern_open() at kern_open+0x18
open() at open+0x14
syscall() at syscall+0x25c
-- syscall (5, FreeBSD ELF64, open) %o7=0x412bfda8 --
userland() at 0x41344808
user trace: trap %o7=0x412bfda8
pc 0x41344808, sp 0x7fdffff99d1
pc 0x120008, sp 0x7fdffff9aa1
pc 0x11216c, sp 0x7fdffff9b61
pc 0x10d32c, sp 0x7fdffffdd11
pc 0x10d9e4, sp 0x7fdffffde01
pc 0x10cf2c, sp 0x7fdffffded1
pc 0x10eea4, sp 0x7fdffffdfa1
pc 0x15a0a0, sp 0x7fdffffe061
pc 0x13024c, sp 0x7fdffffe191
pc 0x103c10, sp 0x7fdffffe281
pc 0x4028ef34, sp 0x7fdffffe341
done

That's just on startup, nothing fancy going on.

My questions are - is it worth spamming the list
with these? If so, what other info needed?

Full dmesg bellow.

Thanks,
--
 Nikolai

GDB: no debug ports present
KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2010 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 9.0-CURRENT #0: Sun Feb 28 23:19:48 EST 2010
    root at moon.ipv6.fetissov.org:/usr/obj/usr/src/sys/GENERIC sparc64
WARNING: WITNESS option enabled, expect reduced performance.
real memory  = 4294967296 (4096 MB)
avail memory = 4174233600 (3980 MB)
cpu0: Sun Microsystems UltraSparc-IIIi Processor (1002.00 MHz CPU)
cpu1: Sun Microsystems UltraSparc-IIIi Processor (1002.00 MHz CPU)
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
ispfw: registered firmware <isp_1000>
ispfw: registered firmware <isp_1040>
ispfw: registered firmware <isp_1040_it>
ispfw: registered firmware <isp_1080>
ispfw: registered firmware <isp_1080_it>
ispfw: registered firmware <isp_12160>
ispfw: registered firmware <isp_12160_it>
ispfw: registered firmware <isp_2100>
ispfw: registered firmware <isp_2200>
ispfw: registered firmware <isp_2300>
ispfw: registered firmware <isp_2322>
ispfw: registered firmware <isp_2400>
ispfw: registered firmware <isp_2400_multi>
ispfw: registered firmware <isp_2500>
ispfw: registered firmware <isp_2500_multi>
kbd0 at kbdmux0
nexus0: <Open Firmware Nexus device>
nexus0: <memory-controller> mem 0x40000000000-0x40000000007 type
memory-controller (no driver attached)
nexus0: <memory-controller> mem 0x40000800000-0x40000800007 type
memory-controller (no driver attached)
pcib0: <Sun Host-PCI bridge> mem
0x4000ff00000-0x4000ff0afff,0x4000fc10000-0x4000fc1701f,0x7f600000000-0x7f6000000ff,0x4000ff80000-0x4000ff8ffff
irq 2035,2032,2033,2036,2019 on nexus0
pcib0: Tomatillo, version 4, IGN 0x1f, bus B, 66MHz
pcib0: DVMA map: 0xc0000000 to 0xdfffffff 65536 entries
pcib0: [FILTER]
pcib0: [FILTER]
pcib0: [FILTER]
pcib0: [FILTER]
pci0: <OFW PCI bus> on pcib0
bge0: <Broadcom BCM5704 A3, ASIC rev. 0x002003> mem
0x200000-0x20ffff,0x110000-0x11ffff at device 2.0 on pci0
miibus0: <MII bus> on bge0
brgphy0: <BCM5704 10/100/1000baseTX PHY> PHY 1 on miibus0
brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT,
1000baseT-FDX, auto
bge0: Ethernet address: 00:03:ba:45:87:2d
bge0: [ITHREAD]
bge1: <Broadcom BCM5704 A3, ASIC rev. 0x002003> mem
0x400000-0x40ffff,0x120000-0x12ffff at device 2.1 on pci0
miibus1: <MII bus> on bge1
brgphy1: <BCM5704 10/100/1000baseTX PHY> PHY 1 on miibus1
brgphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT,
1000baseT-FDX, auto
bge1: Ethernet address: 00:03:ba:45:87:2e
bge1: [ITHREAD]
pcib1: <Sun Host-PCI bridge> mem
0x4000f600000-0x4000f60afff,0x4000f410000-0x4000f41701f,0x7fe00000000-0x7fe000000ff,0x4000f780000-0x4000f78ffff
irq 1970,1968,1969,1972,1953 on nexus0
pcib1: Tomatillo, version 4, IGN 0x1e, bus A, 33MHz
pcib1: DVMA map: 0xc0000000 to 0xdfffffff 65536 entries
pcib1: [FILTER]
pci1: <OFW PCI bus> on pcib1
isab0: <PCI-ISA bridge> at device 7.0 on pci1
isa0: <ISA bus> on isab0
pci1: <old, non-VGA display device> at device 6.0 (no driver attached)
ohci0: <AcerLabs M5237 (Aladdin-V) USB controller> mem 0x1000000-0x1000fff
at device 10.0 on pci1
ohci0: [ITHREAD]
usbus0: <AcerLabs M5237 (Aladdin-V) USB controller> on ohci0
atapci0: <AcerLabs M5229 UDMA100 controller> port
0x900-0x907,0x918-0x91b,0x910-0x917,0x908-0x90b,0x920-0x92f at device 13.0
on pci1
atapci0: [ITHREAD]
atapci0: using PIO transfers above 137GB as workaround for 48bit DMA
access bug, expect reduced performance
ata2: <ATA channel 0> on atapci0
ata2: [ITHREAD]
ata3: <ATA channel 1> on atapci0
ata3: [ITHREAD]
pcib2: <Sun Host-PCI bridge> mem
0x4000e600000-0x4000e60afff,0x4000e410000-0x4000e41701f,0x7ce00000000-0x7ce000000ff,0x4000e780000-0x4000e78ffff
irq 1842,1840,1841,1844,1827 on nexus0
pcib2: Tomatillo, version 4, IGN 0x1c, bus A, 66MHz
pcib2: DVMA map: 0xc0000000 to 0xdfffffff 65536 entries
pcib2: [FILTER]
pcib2: [FILTER]
pcib2: [FILTER]
pcib2: [FILTER]
pci2: <OFW PCI bus> on pcib2
sym0: <1010-66> port 0x900-0x9ff mem 0x100000-0x1003ff,0x102000-0x103fff
at device 2.0 on pci2
sym0: No NVRAM, ID 7, Fast-80, LVD, parity checking
sym0: [ITHREAD]
sym1: <1010-66> port 0xa00-0xaff mem 0x104000-0x1043ff,0x106000-0x107fff
at device 2.1 on pci2
sym1: No NVRAM, ID 7, Fast-80, LVD, parity checking
sym1: [ITHREAD]
pcib3: <Sun Host-PCI bridge> mem
0x4000ef00000-0x4000ef0afff,0x4000ec10000-0x4000ec1701f,0x7c600000000-0x7c6000000ff,0x4000ef80000-0x4000ef8ffff
irq 1907,1904,1905,1908,1893 on nexus0
pcib3: Tomatillo, version 4, IGN 0x1d, bus B, 66MHz
pcib3: DVMA map: 0xc0000000 to 0xdfffffff 65536 entries
pcib3: [FILTER]
pci3: <OFW PCI bus> on pcib3
bge2: <Broadcom BCM5704 A3, ASIC rev. 0x002003> mem
0x200000-0x20ffff,0x110000-0x11ffff at device 2.0 on pci3
miibus2: <MII bus> on bge2
brgphy2: <BCM5704 10/100/1000baseTX PHY> PHY 1 on miibus2
brgphy2:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT,
1000baseT-FDX, auto
bge2: Ethernet address: 00:03:ba:45:87:2f
bge2: [ITHREAD]
bge3: <Broadcom BCM5704 A3, ASIC rev. 0x002003> mem
0x400000-0x40ffff,0x120000-0x12ffff at device 2.1 on pci3
miibus3: <MII bus> on bge3
brgphy3: <BCM5704 10/100/1000baseTX PHY> PHY 1 on miibus3
brgphy3:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT,
1000baseT-FDX, auto
bge3: Ethernet address: 00:03:ba:45:87:30
bge3: [ITHREAD]
nexus0: <syscons> type unknown (no driver attached)
rtc0: <Real-Time Clock> at port 0x70-0x71 on isa0
uart0: <16550 or compatible> at port 0x3f8-0x3ff irq 44 on isa0
uart0: [FILTER]
uart0: console (9600,n,8,1)
uart1: <16550 or compatible> at port 0x2e8-0x2ef irq 44 on isa0
uart1: [FILTER]
Timecounter "tick" frequency 1002000000 Hz quality 10
Timecounters tick every 1.000 msec
usbus0: 12Mbps Full Speed USB v1.0
ugen0.1: <AcerLabs> at usbus0
uhub0: <AcerLabs OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
acd0: DVDROM <DV-28E-N/1.6A> at ata2-master UDMA33
uhub0: 2 ports with 2 removable, self powered
SMP: AP CPU #1 Launched!
da1 at sym0 bus 0 scbus0 target 1 lun 0WARNING: WITNESS option enabled,
expect reduced performance.
da1:
<HITACHI DK32EJ72NSUN72G PQ08> Fixed Direct Access SCSI-3 device
da1: 80.000MB/s transfers (40.000MHz, offset 31, 16bit)
da1: Command Queueing enabled
da1: 70007MB (143374738 512 byte sectors: 255H 63S/T 8924C)
da0 at sym0 bus 0 scbus0 target 0 lun 0
da0: <SEAGATE ST373307LC 0007> Fixed Direct Access SCSI-3 device
da0: 160.000MB/s transfers (80.000MHz DT, offset 62, 16bit)
da0: Command Queueing enabled
da0: 70007MB (143374744 512 byte sectors: 255H 63S/T 8924C)
GEOM: da0: adding VTOC8 information.
GEOM: da1: adding VTOC8 information.
Trying to mount root from ufs:/dev/da0a
lock order reversal:
 1st 0xc1c85fb8 bufwait (bufwait) @ /usr/src/sys/kern/vfs_bio.c:2559
 2nd 0xfffff80002784600 dirhash (dirhash) @
/usr/src/sys/ufs/ufs/ufs_dirhash.c:285
KDB: stack backtrace:
_witness_debugger() at _witness_debugger+0x38
witness_checkorder() at witness_checkorder+0xcf8
_sx_xlock() at _sx_xlock+0x9c
ufsdirhash_acquire() at ufsdirhash_acquire+0x30
ufsdirhash_add() at ufsdirhash_add+0x4
ufs_direnter() at ufs_direnter+0x75c
ufs_makeinode() at ufs_makeinode+0x4dc
ufs_create() at ufs_create+0x40
VOP_CREATE_APV() at VOP_CREATE_APV+0x108
vn_open_cred() at vn_open_cred+0x1fc
vn_open() at vn_open+0x1c
kern_openat() at kern_openat+0x10c
kern_open() at kern_open+0x18
open() at open+0x14
syscall() at syscall+0x25c
-- syscall (5, FreeBSD ELF64, open) %o7=0x412bfda8 --
userland() at 0x41344808
user trace: trap %o7=0x412bfda8
pc 0x41344808, sp 0x7fdffff99d1
pc 0x120008, sp 0x7fdffff9aa1
pc 0x11216c, sp 0x7fdffff9b61
pc 0x10d32c, sp 0x7fdffffdd11
pc 0x10d9e4, sp 0x7fdffffde01
pc 0x10cf2c, sp 0x7fdffffded1
pc 0x10eea4, sp 0x7fdffffdfa1
pc 0x15a0a0, sp 0x7fdffffe061
pc 0x13024c, sp 0x7fdffffe191
pc 0x103c10, sp 0x7fdffffe281
pc 0x4028ef34, sp 0x7fdffffe341
done
lock order reversal:
 1st 0xfffff8000dbb49f8 ufs (ufs) @ /usr/src/sys/kern/vfs_subr.c:2091
 2nd 0xc16573f8 bufwait (bufwait) @ /usr/src/sys/ufs/ffs/ffs_softdep.c:6193
 3rd 0xfffff8000d679718 ufs (ufs) @ /usr/src/sys/kern/vfs_subr.c:2091
KDB: stack backtrace:
_witness_debugger() at _witness_debugger+0x38
witness_checkorder() at witness_checkorder+0xcf8
__lockmgr_args() at __lockmgr_args+0x898
ffs_lock() at ffs_lock+0x88
VOP_LOCK1_APV() at VOP_LOCK1_APV+0x110
_vn_lock() at _vn_lock+0x80
vget() at vget+0x104
vfs_hash_get() at vfs_hash_get+0xdc
ffs_vgetf() at ffs_vgetf+0x20
softdep_sync_metadata() at softdep_sync_metadata+0x6a0
ffs_syncvnode() at ffs_syncvnode+0x424
ffs_truncate() at ffs_truncate+0x454
ufs_direnter() at ufs_direnter+0x920
ufs_mkdir() at ufs_mkdir+0x824
VOP_MKDIR_APV() at VOP_MKDIR_APV+0x108
kern_mkdirat() at kern_mkdirat+0x26c
kern_mkdir() at kern_mkdir+0x14
mkdir() at mkdir+0x10
syscall() at syscall+0x25c
-- syscall (136, FreeBSD ELF64, mkdir) %o7=0x100ce0 --
userland() at 0x40473188
user trace: trap %o7=0x100ce0
pc 0x40473188, sp 0x7fdffffdd71
pc 0x100950, sp 0x7fdffffdeb1
pc 0x40206f34, sp 0x7fdffffdf71
done