US-III crashes on current

[zenxyzzy]

I've been tinkering with my sunblade 1000 for some time, and have run
pretty much all the os's available on it
it's really cool to have those easily swappable fiber channel root disks...

configuration is pretty phat, with 2x900, 4G, 2x73G FC, 500G sata on a
shoehorned in internal bay, and 1 scsi dvd-rom
and 1 ide dvd burner, 2x creator 3d UPA, 1 belkin usb2 card, a promise
4 drive ide card, and a cheap sil3512 sata card.

anyhow, I was tickled pink when 8.0-20090111-SNAP showed up a while
back, and it runs well, with a zfs root, even.
some caveats:

1) the fans run all the time.
2) halt consistently panic's the machine. quite benign, if you think about it:

panic: trap: fast data access mmu miss
cpuid = 0
KDB: enter: panic
[thread pid 1402 tid 100148 ]
Stopped at      kdb_enter+0x80: ta              %xcc, 1
db> where
Tracing pid 1402 tid 100148 td 0xfffff8000448a700
panic() at panic+0x20c
trap() at trap+0x4d0
-- fast data access mmu miss tar=0x14543da000 %o7=0xc034c96c --
callout_lock() at callout_lock+0x40
untimeout() at untimeout+0xc
isp_done() at isp_done+0x140
isp_intr() at isp_intr+0x3eb8
isp_poll() at isp_poll+0x38
xpt_polled_action() at xpt_polled_action+0xc8
dashutdown() at dashutdown+0x16c
boot() at boot+0x858
reboot() at reboot+0x64
syscall() at syscall+0x2e8
-- syscall (55, FreeBSD ELF64, reboot) %o7=0x1013e4 --
userland() at 0x4056af08
user trace: trap %o7=0x1013e4
pc 0x4056af08, sp 0x7fdffffe261
pc 0x100df0, sp 0x7fdffffe321
pc 0x402066f4, sp 0x7fdffffe3e1

3) no X
4) no sound
5) annoying lock order reversals.
6) under extreme loads (load av == 10) possibly a hang or two.

so, since I want to contribute some data, I build a kernel from the
SNAP's source, and it works just as well, even with
the lock instrumentation removed.

so, I pull a current source tree and build it. oops. no go.  1100+
files changed in those 2 months; how to find the culprit?

it panics long before probing devices, using the generic config file:

BOOM:

Hit [Enter] to boot immediately, or any other key for command prompt.
Booting [/boot/kernel/kernel]...
jumping to kernel entry at 0xc0080000.
GDB: no debug ports present
KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2009 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.0-CURRENT #0: Sun Mar 22 09:47:54 CDT 2009
    root at ra.zen-room.org:/usr/src/sys/sparc64/compile/SAFE
WARNING: WITNESS option enabled, expect reduced performance.
real memory  = 4294967296 (4096 MB)
panic: vm_phys_paddr_to_vm_page: paddr 0xfd81a000 is not in any segment
cpuid = 0
KDB: enter: panic
[thread pid 0 tid 0 ]
Stopped at      kdb_enter+0x80: ta              %xcc, 1
db> where
Tracing pid 0 tid 0 td 0xc08ad670
panic() at panic+0x20c
vm_phys_paddr_to_vm_page() at vm_phys_paddr_to_vm_page+0x84
pmap_remove_tte() at pmap_remove_tte+0x80
pmap_enter_locked() at pmap_enter_locked+0x204
pmap_enter() at pmap_enter+0x64
vm_fault() at vm_fault+0x17ac
vm_fault_wire() at vm_fault_wire+0x3c
vm_map_wire() at vm_map_wire+0x26c
kmem_alloc() at kmem_alloc+0x1b4
vm_ksubmap_init() at vm_ksubmap_init+0x74
cpu_startup() at cpu_startup+0xc4
mi_startup() at mi_startup+0x18c
btext() at btext+0x30

anybody got any better source than 8.0-20090111-SNAP?  Those 1100 file
changes look pretty daunting.

until then, I'll just work on X.

oh, yeah: the obligatory:

GDB: no debug ports present
KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2009 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.0-20090111-SNAP #0: Sat Mar 21 13:19:27 CDT 2009
    root at ra.zen-room.org:/usr/src/sys/sparc64/compile/RA2
real memory  = 4294967296 (4096 MB)
avail memory = 4174684160 (3981 MB)
cpu0: Sun Microsystems UltraSparc-III+ Processor (900.00 MHz CPU)
cpu1: Sun Microsystems UltraSparc-III+ Processor (900.00 MHz CPU)
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
This module (opensolaris) contains code covered by the
Common Development and Distribution License (CDDL)
see http://opensolaris.org/os/licensing/opensolaris_license/
ispfw: registered firmware <isp_1000>
ispfw: registered firmware <isp_1040>
ispfw: registered firmware <isp_1040_it>
ispfw: registered firmware <isp_1080>
ispfw: registered firmware <isp_1080_it>
ispfw: registered firmware <isp_12160>
ispfw: registered firmware <isp_12160_it>
ispfw: registered firmware <isp_2100>
ispfw: registered firmware <isp_2200>
ispfw: registered firmware <isp_2300>
ispfw: registered firmware <isp_2322>
ispfw: registered firmware <isp_2400>
kbd0 at kbdmux0
nexus0: <Open Firmware Nexus device>
nexus0: <memory-controller> mem 0x40000400000-0x40000400047 type
memory-controller (no driver attached)
nexus0: <memory-controller> mem 0x40000c00000-0x40000c00047 type
memory-controller (no driver attached)
pcib0: <Sun Host-PCI bridge> mem
0x40004700000-0x40004717fff,0x40004410000-0x4000441004f,0x7ffee000000-0x7ffee0000ff
irq 563,560,561,564,550 on nexus0
pcib0: Schizo, version 4, IGN 0x8, bus B, 33MHz
pcib0: DVMA map: 0xc0000000 to 0xffffffff
pcib0: [FILTER]
pcib0: [FILTER]
pcib0: [FILTER]
pcib0: [FILTER]
pci0: <OFW PCI bus> on pcib0
ebus0: <PCI-EBus3 bridge> mem
0x7d000000-0x7dffffff,0x7e000000-0x7e7fffff at device 5.0 on pci0
ebus0: <flashprom> addr 0-0x1fffff (no driver attached)
ebus0: <i2c> addr 0x10000002e-0x10000002f,0x10000002d irq 35 (no
driver attached)
ebus0: <bbc> addr 0x100000000-0x1000fffff (no driver attached)
ebus0: <ppm> addr
0x10000000e-0x100000028,0x100728000-0x100728003,0x10030002e-0x10030002f,0x100300600-0x100300607
(no driver attached)
ebus0: <i2c> addr 0x100000030-0x100000031 irq 35 (no driver attached)
ebus0: <beep> addr 0x100000032-0x100000037 (no driver attached)
ebus0: <audio> addr
0x100200000-0x1002000ff,0x100702000-0x10070200f,0x100704000-0x10070400f,0x100722000-0x100722003
irq 32,33 (no driver attached)
rtc0: <Real-Time Clock> addr 0x100300070-0x100300071 irq 36 on ebus0
ebus0: <gpio> addr 0x100300600-0x100300607 (no driver attached)
ebus0: <pmc> addr 0x100300700-0x100300701 (no driver attached)
ebus0: <floppy> addr
0x1003023f0-0x1003023f7,0x100706000-0x10070600f,0x100720000-0x100720003
irq 37 (no driver attached)
ebus0: <parallel> addr
0x100300278-0x100300287,0x10030002e-0x10030002f,0x100700000-0x10070000f
irq 28 (no driver attached)
scc0: <Siemens SAB 82532 dual channel SCC> addr
0x100400000-0x10040007f irq 34 on ebus0
scc0: [FILTER]
uart0: <SAB 82532 v3.2, channel A> on scc0
uart0: [FILTER]
uart0: CTS oflow
uart0: console (38400,n,8,1)
uart1: <SAB 82532 v3.2, channel B> on scc0
uart1: [FILTER]
uart1: CTS oflow
gem0: <Sun ERI 10/100 Ethernet> mem 0x100000-0x11ffff at device 5.1 on pci0
miibus0: <MII bus> on gem0
ukphy0: <Generic IEEE 802.3u media interface> PHY 1 on miibus0
ukphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
gem0: 2kB RX FIFO, 2kB TX FIFO
gem0: Ethernet address: 00:03:ba:12:5d:81
gem0: [ITHREAD]
fwohci0: <Sun PCIO-2> mem 0x120000-0x1207ff,0x122000-0x1227ff at
device 5.2 on pci0
fwohci0: [FILTER]
fwohci0: OHCI version 1.0 (ROM=0)
fwohci0: No. of Isochronous channels is 4.
fwohci0: EUI64 00:03:ba:ff:fe:12:5d:81
fwohci0: Phy 1394a available S400, 4 ports.
fwohci0: Link S400, max_rec 2048 bytes.
firewire0: <IEEE1394(FireWire) bus> on fwohci0
fwe0: <Ethernet over FireWire> on firewire0
if_fwe0: Fake Ethernet address: 02:03:ba:12:5d:81
fwe0: Ethernet address: 02:03:ba:12:5d:81
fwip0: <IP over FireWire> on firewire0
fwip0: Firewire address: 00:03:ba:ff:fe:12:5d:81 @ 0xfffe00000000,
S400, maxrec 2048
sbp0: <SBP-2/SCSI over FireWire> on firewire0
dcons_crom0: <dcons configuration ROM> on firewire0
dcons_crom0: bus_addr 0xc1128000
fwohci0: Initiate bus reset
fwohci0: BUS reset
fwohci0: node_id=0xc800ffc0, gen=1, CYCLEMASTER mode
ohci0: <Sun PCIO-2 USB controller> mem 0x1000000-0x1007fff at device 5.3 on pci0
ohci0: [GIANT-LOCKED]
ohci0: [ITHREAD]
usb0: OHCI version 1.0, legacy support
usb0: <Sun PCIO-2 USB controller> on ohci0
usb0: USB revision 1.0
uhub0: <(0x108e) OHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb0
uhub0: 4 ports with 4 removable, self powered
sym0: <875> port 0x300-0x3ff mem 0x124000-0x1240ff,0x126000-0x126fff
at device 6.0 on pci0
sym0: No NVRAM, ID 7, Fast-20, SE, parity checking
sym0: [ITHREAD]
sym1: <875> port 0x400-0x4ff mem 0x128000-0x1280ff,0x12a000-0x12afff
at device 6.1 on pci0
sym1: No NVRAM, ID 7, Fast-20, SE, parity checking
sym1: [ITHREAD]
ohci1: <NEC uPD 9210 USB controller> mem 0x12c000-0x12cfff at device 1.0 on pci0
ohci1: [GIANT-LOCKED]
ohci1: [ITHREAD]
usb1: OHCI version 1.0, legacy support
usb1: <NEC uPD 9210 USB controller> on ohci1
usb1: USB revision 1.0
uhub1: <NEC OHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb1
uhub1: 3 ports with 3 removable, self powered
ohci2: <NEC uPD 9210 USB controller> mem 0x12e000-0x12efff at device 1.1 on pci0
ohci2: [GIANT-LOCKED]
ohci2: [ITHREAD]
usb2: OHCI version 1.0, legacy support
usb2: <NEC uPD 9210 USB controller> on ohci2
usb2: USB revision 1.0
uhub2: <NEC OHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb2
uhub2: 2 ports with 2 removable, self powered
ehci0: <NEC uPD 720100 USB 2.0 controller> mem 0x130000-0x1300ff at
device 1.2 on pci0
ehci0: [GIANT-LOCKED]
ehci0: [ITHREAD]
usb3: EHCI version 1.0
usb3: companion controllers, 3 ports each: usb1 usb2
usb3: <NEC uPD 720100 USB 2.0 controller> on ehci0
usb3: USB revision 2.0
uhub3: <NEC EHCI root hub, class 9/0, rev 2.00/1.00, addr 1> on usb3
uhub3: 5 ports with 5 removable, self powered
atapci0: <Promise PDC20246 UDMA33 controller> port
0x11f0-0x11f7,0x13f4-0x13f7,0x1170-0x1177,0x1374-0x1377,0x6800-0x681f
at device 3.0 on pci0
atapci0: [ITHREAD]
ata2: <ATA channel 0> on atapci0
ata2: [ITHREAD]
ata3: <ATA channel 1> on atapci0
ata3: [ITHREAD]
pcib1: <Sun Host-PCI bridge> mem
0x40004600000-0x40004617fff,0x40004410000-0x4000441004f,0x7ffec000000-0x7ffec0000ff
irq 562,560,561,564,551 on nexus0
pcib1: Schizo, version 4, IGN 0x8, bus A, 66MHz
Timecounter "pcib1" frequency 150000000 Hz quality 100
pcib1: DVMA map: 0xc0000000 to 0xffffffff
pcib1: [FILTER]
pci1: <OFW PCI bus> on pcib1
isp0: <Qlogic ISP 2200 PCI FC-AL Adapter> port 0x300-0x3ff mem
0x100000-0x100fff at device 4.0 on pci1
isp0: [ITHREAD]
isp0: Board Type 2200, Chip Revision 0x5, loaded F/W Revision 2.2.6
isp0: invalid NVRAM header
isp0: invalid NVRAM header
atapci1: <SiI 3512 SATA150 controller> port
0x400-0x407,0x408-0x40b,0x410-0x417,0x40c-0x40f,0x420-0x42f mem
0x102000-0x1021ff at device 1.0 on pci1
atapci1: [ITHREAD]
ata4: <ATA channel 0> on atapci1
ata4: [ITHREAD]
ata5: <ATA channel 1> on atapci1
ata5: [ITHREAD]
upa0: <UPA bridge> mem
0x40004480000-0x4000448001f,0x40004706000-0x4000470600f,0x40004708000-0x4000470800f
on nexus0
creator0: <Creator3D> mem
0x1c000000000-0x1c0000003ff,0x1c000400000-0x1c0005fffff,0x1c000600000-0x1c0007fffff,0x1c001000000-0x1c0013fffff,0x1c001400000-0x1c0017fffff,0x1c001800000-0x1c001bfffff,0x1c001c00000-0x1c001ffffff,0x1c002000000-0x1c002ffffff,0x1c003000000-0x1c003ffffff,0x1c004000000-0x1c0043fffff,0x1c004400000-0x1c0047fffff,0x1c004800000-0x1c004bfffff,0x1c004c00000-0x1c004ffffff,0x1c005000000-0x1c005ffffff,0x1c006000000-0x1c007ffffff,0x1c009000000-0x1c0097fffff,0x1c009800000-0x1c009ffffff,0x1c00a000000-0x1c00affffff,0x1c00b000000-0x1c00b7fffff,0x1c00b800000-0x1c00bffffff,0x1c00c000000-0x1c00c3fffff,0x1c00c800000-0x1c00cffffff,0x1c00d000000-0x1c00d7fffff,0x1c00d800000-0x1c00dffffff
irq 554 on upa0
creator0: resolution 1152x900
creator1: <Creator3D> mem
0x1c200000000-0x1c2000003ff,0x1c200400000-0x1c2005fffff,0x1c200600000-0x1c2007fffff,0x1c201000000-0x1c2013fffff,0x1c201400000-0x1c2017fffff,0x1c201800000-0x1c201bfffff,0x1c201c00000-0x1c201ffffff,0x1c202000000-0x1c202ffffff,0x1c203000000-0x1c203ffffff,0x1c204000000-0x1c2043fffff,0x1c204400000-0x1c2047fffff,0x1c204800000-0x1c204bfffff,0x1c204c00000-0x1c204ffffff,0x1c205000000-0x1c205ffffff,0x1c206000000-0x1c207ffffff,0x1c209000000-0x1c2097fffff,0x1c209800000-0x1c209ffffff,0x1c20a000000-0x1c20affffff,0x1c20b000000-0x1c20b7fffff,0x1c20b800000-0x1c20bffffff,0x1c20c000000-0x1c20c3fffff,0x1c20c800000-0x1c20cffffff,0x1c20d000000-0x1c20d7fffff,0x1c20d800000-0x1c20dffffff
irq 555 on upa0
creator1: resolution 1152x900
schppm0: <Schizo power management> mem 0x40004410050-0x40004410057 on nexus0
syscons0: <System console> on nexus0
syscons0: Unknown <16 virtual consoles, flags=0x100>
ukbd0: <ATEN UC-10KM V1.3.121, class 0/0, rev 1.10/1.00, addr 2> on uhub0
kbd1 at ukbd0
ums0: <ATEN UC-10KM V1.3.121, class 0/0, rev 1.10/1.00, addr 2> on uhub0
ums0: 5 buttons and Z dir.
WARNING: ZFS is considered to be an experimental feature in FreeBSD.
Timecounter "tick" frequency 900000000 Hz quality 10
Timecounters tick every 1.000 msec
firewire0: 1 nodes, maxhop <= 0, cable IRM = 0 (me)
firewire0: bus manager 0 (me)
ZFS filesystem version 13
ZFS storage pool version 13
acd0: DVDR <ATAPI DVD A LH-1P20A/KG45> at ata2-master PIO4
ad0: 476940MB <Seagate ST3500320AS SD35> at ata5-master SATA150
Waiting 5 seconds for SCSI devices to settle
SMP: AP CPU #1 Launched!
cd0 at sym0 bus 0 target 6 lun 0
cd0: <TOSHIBA DVD-ROM SD-M1401 1009> Removable CD-ROM SCSI-2 device
cd0: 3.300MB/s transfers
cd0: cd present [1002840 x 512 byte records]
da0 at isp0 bus 0 target 0 lun 0
da0: <FUJITSU MAT3073FC 0108> Fixed Direct Access SCSI-4 device
da0: 100.000MB/s transfers
da0: Command Queueing Enabled
da0: 70136MB (143638992 512 byte sectors: 255H 63S/T 8941C)
da2 at isp0 bus 0 target 1 lun 0
da2: <FUJITSU MAT3073FC 0107> Fixed Direct Access SCSI-4 device
da2: 100.000MB/s transfers WWNN 0x500000e01111f280 WWPN
0x500000e01111f281 PortID 0xe4
da2: Command Queueing Enabled
da2: 70136MB (143638992 512 byte sectors: 255H 63S/T 8941C)
GEOM: da0: adding VTOC8 information.
GEOM: cd0: invalid VTOC8 label.
GEOM_LABEL: Label for provider cd0 is iso9660/FreeBSD_Install.
GEOM: iso9660/FreeBSD_Install: invalid VTOC8 label.
Trying to mount root from zfs:rpool