Kernel panics in ahc during load with stable built 18th Feb

Tony Frank tfrank at optushome.com.au
Wed Feb 18 21:23:42 PST 2004


Hi all,

As per the subject I seem to be getting kernel panics in ahc 
driver since upgrading my kernel & world to -stable.
This occurs specifically when writing high volume of files to
vinum raid5 volume spanning 4 scsi drives connected to Adaptec 2940 PCI
controller.

The fault appears to be reproducable - every time I try to extract
a tar file containing a copy of /usr/obj from another system.

vinum init and a lot of benchmarking (rawio and bonnie) work fine on the
volume. 

dmesg lines:
ahc0: <Adaptec 2940 Ultra SCSI adapter> port 0xb400-0xb4ff mem 0xe0800000-0xe080
0fff irq 10 at device 11.0 on pci0

Custom kernel is configured with:
options 	AHC_ALLOW_MEMIO 

I setup a serial console and rebuilt kernel to include debugging bits.

Fatal trap 12: page fault while in kernel mode
fault virtual address   = 0x5c
fault code              = supervisor read, page not present
instruction pointer     = 0x8:0xc015cab2
stack pointer           = 0x10:0xc02e2b58
frame pointer           = 0x10:0xc02e2b68
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, def32 1, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = Idle
interrupt mask          = cam
kernel: type 12 trap, code=0
Stopped at      ahc_done+0xc2:  pushl   0x5c(%ebx)

db> trace
ahc_done(c0f9a200,c0fb53c0) at ahc_done+0xc2
ahc_run_qoutfifo(c0f9a200) at ahc_run_qoutfifo+0xf1
ahc_platform_intr(c0f9a200,0,c02e2bf8,c027ab82,c0322458) at ahc_platform_intr+0x
174
add_interrupt_randomness(c0322458,0,400010,c0300010,c0300010) at add_interrupt_r
andomness+0xe
Xresume10() at Xresume10+0x2b
--- interrupt, eip = 0xc027fa46, esp = 0xc02e2bf0, ebp = 0xc02e2bf8 ---
cpu_idle(e,633,2,80f9ff,0) at cpu_idle+0xe
idle_loop() at idle_loop+0x1d

Follows is the full log of boot and panic and my feeble attempts to do get
some further info. (ps an trace)

Any suggestions/pointers?

Thanks,

Tony


Console: serial port
BIOS drive A: is disk0
BIOS drive C: is disk1
BIOS drive D: is disk2
BIOS drive E: is disk3
BIOS drive F: is disk4
BIOS drive G: is disk5
BIOS drive H: is disk6
BIOS 639kB/130036kB available memory

FreeBSD/i386 bootstrap loader, Revision 0.8
(tony at raider.home.local, Thu Feb 19 13:52:39 EST 2004)
Loading /boot/defaults/loader.conf
/kernel text=0x1dfc00 data=0x27ea0+0x22994 syms=[0x4+0x2c6f0+0x4+0x345f2]
/modules/vinum.ko text=0x14e20 data=0x3d0+0xace88 syms=[0x4+0x1250+0x4+0xc37]

Hit [Enter] to boot immediately, or any other key for command prompt.
Booting [kernel]...
Copyright (c) 1992-2003 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD 4.9-STABLE #2: Thu Feb 19 14:24:16 EST 2004
    tony at marvin.home.local:/data/ad3/obj/usr/src/sys/RAIDER
Timecounter "i8254"  frequency 1193182 Hz
CPU: Pentium II/Pentium II Xeon/Celeron (200.46-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0x633  Stepping = 3
  Features=0x80f9ff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,SEP,MTRR,PGE,MCA,CMOV,MMX
>
real memory  = 134205440 (131060K bytes)
config> en apm0
avail memory = 126218240 (123260K bytes)
Preloaded elf kernel "kernel" at 0xc0453000.
Preloaded userconfig_script "/boot/kernel.conf" at 0xc045309c.
Preloaded elf module "vinum.ko" at 0xc04530ec.
VESA: v2.0, 2048k memory, flags:0x0, mode table:0xc0322ac2 (1000022)
VESA: ATI MACH64
Pentium Pro MTRR support enabled
md0: Malloc disk
Using $PIR table, 6 entries at 0xc00f0b60
apm0: <APM BIOS> on motherboard
apm0: found APM BIOS v1.2, connected at v1.2
npx0: <math processor> on motherboard
npx0: INT 16 interface
pcib0: <Host to PCI bridge> on motherboard
pci0: <PCI bus> on pcib0
agp0: <VIA 82C691 (Apollo Pro) host to PCI bridge> mem 0xe4000000-0xe7ffffff at
device 0.0 on pci0
pcib2: <VIA 82C598MVP (Apollo MVP3) PCI-PCI (AGP) bridge> at device 1.0 on pci0
pci1: <PCI bus> on pcib2
isab0: <VIA 82C596B PCI-ISA bridge> at device 4.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <VIA 82C596 ATA33 controller> port 0xd800-0xd80f at device 4.1 on pci0
ata0: at 0x1f0 irq 14 on atapci0
ata1: at 0x170 irq 15 on atapci0
pci0: <VIA 83C572 USB controller> at 4.2
pci0: <unknown card> (vendor=0x1106, dev=0x3050) at 4.3
pci0: <ATI Mach64-GT graphics accelerator> at 9.0
fxp0: <Intel 82559 Pro/100 Ethernet> port 0xb800-0xb83f mem 0xe1000000-0xe10ffff
f,0xe1800000-0xe1800fff irq 3 at device 10.0 on pci0
fxp0: Ethernet address 00:04:ac:e5:d3:59
inphy0: <i82555 10/100 media interface> on miibus0
inphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
ahc0: <Adaptec 2940 Ultra SCSI adapter> port 0xb400-0xb4ff mem 0xe0800000-0xe080
0fff irq 10 at device 11.0 on pci0
aic7880: Ultra Wide Channel A, SCSI Id=7, 16/253 SCBs
fxp1: <Intel 82559 Pro/100 Ethernet> port 0xb000-0xb03f mem 0xdf800000-0xdf8ffff
f,0xe0000000-0xe0000fff irq 11 at device 12.0 on pci0
fxp1: Ethernet address 00:03:47:3a:43:63
inphy1: <i82555 10/100 media interface> on miibus1
inphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
pcib1: <Host to PCI bridge> on motherboard
pci2: <PCI bus> on pcib1
orm0: <Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xc97ff,0xcc000-0xd07ff,0xd
4000-0xd57ff on isa0
pmtimer0 on isa0
fdc0: <NEC 72065B or clone> at port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on isa0
fdc0: FIFO enabled, 8 bytes threshold
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> flags 0x1 irq 1 on atkbdc0
kbd0 at atkbd0
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: model MouseMan+, device ID 0
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x100>
sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
sio0: type 16550A, console
sio1: configured irq 3 not in bitmap of probed irqs 0
ppc0: parallel port not found.
ipfw2 initialized, divert enabled, rule-based forwarding enabled, default to den
y, logging unlimited
DUMMYNET initialized (011031)
IPsec: Initialized Security Association Processing.
ad0: 8063MB <FUJITSU MPE3084AE> [16383/16/63] at ata0-master UDMA33
ad2: 3093MB <FUJITSU MPC3032AT> [6704/15/63] at ata1-master UDMA33
acd0: CDROM <CRD-8160B> at ata1-slave PIO4
Waiting 9 seconds for SCSI devices to settle
vinum: loadcd0 at ata1 bus 0 target 1 lun 0
cd0: <GoldStar CD-ROM CRD-8160B 3.14> Removable CD-ROM SCSI-0 device
cd0: 16.000MB/s transfers
cd0: Attempt to query device size failed: NOT READY, Medium not present
ed
da2 at ahc0 bus 0 target 2 lun 0
da2: <IBM DFHSS4W 4D4D> Fixed Direct Access SCSI-2 device
da2: 20.000MB/s transfers (10.000MHz, offset 8, 16bit), Tagged Queueing Enabled
da2: 4303MB (8813870 512 byte sectors: 255H 63S/T 548C)
da3 at ahc0 bus 0 target 3 lun 0
da3: <IBM DFHSS4W 4D4D> Fixed Direct Access SCSI-2 device
da3: 20.000MB/s transfers (10.000MHz, offset 8, 16bit), Tagged Queueing Enabled
da3: 4303MB (8813870 512 byte sectors: 255H 63S/T 548C)
da1 at ahc0 bus 0 target 1 lun 0
da1: <IBM DFHSS4W 4D4D> Fixed Direct Access SCSI-2 device
da1: 20.000MB/s transfers (10.000MHz, offset 8, 16bit), Tagged Queueing Enabled
da1: 4303MB (8813870 512 byte sectors: 255H 63S/T 548C)
da0 at ahc0 bus 0 target 0 lun 0
da0: <IBM DFHSS4W 4D4D> Fixed Direct Access SCSI-2 device
da0: 20.000MB/s transfers (10.000MHz, offset 8, 16bit), Tagged Queueing Enabled
da0: 4303MB (8813870 512 byte sectors: 255H 63S/T 548C)
vinum: reading configuration from /dev/da3s1h
vinum: updating configuration from /dev/da2s1h
vinum: updating configuration from /dev/da1s1h
vinum: updating configuration from /dev/da0s1h
vinum: updating configuration from /dev/ad2s1h
vinum: updating configuration from /dev/ad0s1h
vinum: using volume root for root device
Mounting root from ufs:/dev/vinum/root
dumpon: crash dumps to /dev/ad0s1b (116, 131073)
swapon: adding /dev/vinum/swap as swap device
Automatic boot in progress...
/dev/vinum/root: FILESYSTEM CLEAN; SKIPPING CHECKS
/dev/vinum/root: clean, 190920 free (208 frags, 23839 blocks, 0.1% fragmentation
)
/dev/vinum/tmp: FILESYSTEM CLEAN; SKIPPING CHECKS
/dev/vinum/tmp: clean, 254061 free (21 frags, 31755 blocks, 0.0% fragmentation)
/dev/vinum/usr: FILESYSTEM CLEAN; SKIPPING CHECKS
/dev/vinum/usr: clean, 892204 free (3044 frags, 111145 blocks, 0.1% fragmentatio
n)
/dev/vinum/data: FILESYSTEM CLEAN; SKIPPING CHECKS
/dev/vinum/data: clean, 16532723 free (211 frags, 2066564 blocks, 0.0% fragmenta
tion)
/dev/vinum/var: FILESYSTEM CLEAN; SKIPPING CHECKS
/dev/vinum/var: clean, 253586 free (58 frags, 31691 blocks, 0.0% fragmentation)
Doing initial network setup: hostname.
fxp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        inet 192.168.3.7 netmask 0xffffff00 broadcast 192.168.3.255
        ether 00:04:ac:e5:d3:59
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
        inet 127.0.0.1 netmask 0xff000000
Flushed all rules.
00100 allow ip from any to any via lo0
00200 deny ip from any to 127.0.0.0/8
00300 deny ip from 127.0.0.0/8 to any
65000 allow ip from any to any
Firewall rules loaded, starting divert daemons:.
Additional routing options: TCP keepalive=YES.
Routing daemons:.
Clearing /tmp:.
Additional daemons: syslogd.
Checking for core dump: savecore: no core dump
Doing additional network setup: ntpdate ntpd.
Starting final network daemons: nfsiod NFS access cache time=2.
ELF ldconfig path: /usr/lib /usr/lib/compat
a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout
Starting standard daemons: cron sshd.
Initial rc.i386 initialization: apm.
entropy IRQs: 1 5 6 10 14 15.
Configuring syscons: blanktime screensaver.
Additional ABI support:.
Local package initialization:.
Additional TCP options:.

Thu Feb 19 15:06:46 EST 2004

FreeBSD/i386 (raider.home.local) (ttyd0)

login: Feb 19 15:15:29 raider ntpd[101]: kernel time discipline status change 20
41


Fatal trap 12: page fault while in kernel mode
fault virtual address   = 0x5c
fault code              = supervisor read, page not present
instruction pointer     = 0x8:0xc015cab2
stack pointer           = 0x10:0xc02e2b58
frame pointer           = 0x10:0xc02e2b68
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, def32 1, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = Idle
interrupt mask          = cam
kernel: type 12 trap, code=0
Stopped at      ahc_done+0xc2:  pushl   0x5c(%ebx)
db> ps
  pid   proc     addr    uid  ppid  pgrp  flag stat wmesg   wchan   cmd
  173 c7f81220 ccdbd000 1001   164   173 004006  3   biowr c38a44a0 tar
  164 c7f813c0 ccdb9000 1001   163   164 2004086  3   pause ccdb9260 tcsh
  163 c7f81560 ccdb2000 1001   161   118 000184  3  select c0324168 sshd
  161 c7f81a40 ccd9e000    0   118   118 000184  3  sbwait cc5bf608 sshd
  158 c7f818a0 ccda3000 1001   153   158 004186  3   ttyin c10e3910 systat
  155 c7f81700 ccdaf000 1001   152   155 004186  3  select c0324168 top
  154 c7f82dc0 ccd30000    0     1   154 004086  3   ttyin c105b810 getty
  153 c7f825a0 ccd71000    0     1   153 004186  3    wait c7f825a0 login
  152 c7f82f60 ccd28000    0     1   152 004186  3    wait c7f82f60 login
  118 c7f81be0 ccd92000    0     1   118 000184  3  select c0324168 sshd
  116 c7f81d80 ccd85000    0     1   116 000084  3  nanslp c030d2b0 cron
  108 c7f81f20 ccd81000    0     1   104 000084  3  nfsidl c032a6ac nfsiod
  107 c7f820c0 ccd7d000    0     1   104 000084  3  nfsidl c032a6a8 nfsiod
  106 c7f82260 ccd79000    0     1   104 000084  3  nfsidl c032a6a4 nfsiod
  105 c7f82400 ccd75000    0     1   104 000084  3  nfsidl c032a6a0 nfsiod
  101 c7f82c20 ccd38000    0     1   101 000084  3  select c0324168 ntpd
   96 c7f82a80 ccd3c000    0     1    96 000084  3  select c0324168 syslogd
   69 c7f82740 ccd45000    0     1    69 000084  3  select c0324168 dhclient
   29 c7f828e0 ccd40000    0     1    29 2000084  3   pause ccd40260 adjkerntz
    9 c7f83100 ccac0000    0     0     0 000204  3  vlruwt c7f83100 vnlru
    8 c7f832a0 ccabd000    0     0     0 000204  3  vrlock c104c010 syncer
    7 c7f83440 ccaba000    0     0     0 000204  3  vrlock c104c030 bufdaemon
    6 c7f835e0 ccab7000    0     0     0 000204  3  psleep c031b260 vmdaemon
    5 c7f83780 ccab4000    0     0     0 000204  3  psleep c02ffef8 pagedaemon
    4 c7f83920 cc5b9000    0     0     0 000204  3    idle c0f9a200 aic_recovery
0
    3 c7f83ac0 cc5b6000    0     0     0 000204  3    idle c0f9a200 aic_recovery
0
    2 c7f83c60 c856b000    0     0     0 000204  3   tqthr c0324164 taskqueue
    1 c7f83e00 c7f88000    0     0     1 004284  3    wait c7f83e00 init
    0 c0323460 c0473000    0     0     0 000204  3   sched c0323460 swapper
db> trace
ahc_done(c0f9a200,c0fb53c0) at ahc_done+0xc2
ahc_run_qoutfifo(c0f9a200) at ahc_run_qoutfifo+0xf1
ahc_platform_intr(c0f9a200,0,c02e2bf8,c027ab82,c0322458) at ahc_platform_intr+0x
174
add_interrupt_randomness(c0322458,0,400010,c0300010,c0300010) at add_interrupt_r
andomness+0xe
Xresume10() at Xresume10+0x2b
--- interrupt, eip = 0xc027fa46, esp = 0xc02e2bf0, ebp = 0xc02e2bf8 ---
cpu_idle(e,633,2,80f9ff,0) at cpu_idle+0xe
idle_loop() at idle_loop+0x1d
db>


More information about the freebsd-stable mailing list