kern/55191: frequent kernel page faults on MCLGET()s

[Matt Fredette]

>Number:         55191
>Category:       kern
>Synopsis:       frequent kernel page faults on MCLGET()s
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sat Aug 02 12:20:06 PDT 2003
>Closed-Date:
>Last-Modified:
>Originator:     Matt Fredette
>Release:        FreeBSD 4.8-RELEASE i386
>Organization:
self
>Environment:
System: FreeBSD no-quarter.home 4.8-RELEASE FreeBSD 4.8-RELEASE #1: Thu Jul 31 16:36:00 EDT 2003 fredette at no-quarter.home:/data/home/fredette/sys/compile/NO-QUARTER i386

Copyright (c) 1992-2003 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD 4.8-RELEASE #1: Thu Jul 31 16:36:00 EDT 2003
    fredette at no-quarter.home:/data/home/fredette/sys/compile/NO-QUARTER
Timecounter "i8254"  frequency 1193182 Hz
CPU: Pentium II/Pentium II Xeon/Celeron (233.14-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0x634  Stepping = 4
  Features=0x80f9ff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,SEP,MTRR,PGE,MCA,CMOV,MMX>
real memory  = 301989888 (294912K bytes)
avail memory = 290025472 (283228K bytes)
Preloaded elf kernel "kernel" at 0xc035b000.
Preloaded userconfig_script "/boot/kernel.conf" at 0xc035b09c.
Pentium Pro MTRR support enabled
md0: Malloc disk
Using $PIR table, 7 entries at 0xc00fde80
npx0: <math processor> on motherboard
npx0: INT 16 interface
pcib0: <Intel 82443LX (440 LX) host to PCI bridge> on motherboard
pci0: <PCI bus> on pcib0
agp0: <Intel 82443LX (440 LX) host to PCI bridge> mem 0xf8000000-0xfbffffff at device 0.0 on pci0
pcib1: <Intel 82443LX (440 LX) PCI-PCI (AGP) bridge> at device 1.0 on pci0
pci1: <PCI bus> on pcib1
isab0: <Intel 82371AB PCI to ISA bridge> at device 7.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel PIIX4 ATA33 controller> port 0xfcd0-0xfcdf at device 7.1 on pci0
ata0: at 0x1f0 irq 14 on atapci0
ata1: at 0x170 irq 15 on atapci0
uhci0: <Intel 82371AB/EB (PIIX4) USB controller> port 0xfce0-0xfcff irq 9 at device 7.2 on pci0
usb0: <Intel 82371AB/EB (PIIX4) USB controller> on uhci0
usb0: USB revision 1.0
uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
chip0: <Intel 82371AB Power management controller> port 0x7000-0x700f at device 7.3 on pci0
pci0: <Rendition Verite V2000 SVGA controller> at 14.0 irq 11
xl0: <3Com 3c905C-TX Fast Etherlink XL> port 0xfc00-0xfc7f mem 0xfedbfc00-0xfedbfc7f irq 9 at device 16.0 on pci0
xl0: Ethernet address: 00:50:da:ce:e9:b7
miibus0: <MII bus> on xl0
xlphy0: <3c905C 10/100 internal PHY> on miibus0
xlphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
orm0: <Option ROMs> at iomem 0xc8000-0xc87ff,0xe0000-0xe3fff,0xe4000-0xeffff on isa0
fdc0: <NEC 72065B or clone> at port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on isa0
fdc0: FIFO enabled, 8 bytes threshold
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> flags 0x1 irq 1 on atkbdc0
kbd0 at atkbd0
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x100>
sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
sio0: type 16550A, console
sio1: configured irq 3 not in bitmap of probed irqs 0
ppc0: parallel port not found.
ad0: 3244MB <SAMSUNG VG33402A (3.40GB)> [6591/16/63] at ata0-master UDMA33
acd0: CDROM <FX162T4> at ata0-slave PIO3
Mounting root from ufs:/dev/ad0s1a

>Description:

Once or more per day this machine gets kernel page faults while running 
galeon from the galeon2-1.3.2_1 package.

I have traces of five crashes.  Four of them look like this:

Fatal trap 12: page fault while in kernel mode
fault virtual address	= 0xc0beb948
fault code		= supervisor write, page not present
instruction pointer	= 0x8:0xc0180567
stack pointer	        = 0x10:0xd29b8e2c
frame pointer	        = 0x10:0xd29b8e50
code segment		= base 0x0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, def32 1, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 127 (galeon-bin)
interrupt mask		= net tty 
kernel: type 12 trap, code=0
Stopped at      sosend+0x443:   incb    0(%eax,%edx,1)
db> tr
sosend(d1c2dd00,0,d29b8f08,0,0) at sosend+0x443
soo_write(c1515880,d29b8f08,c146fc00,0,cd10f8a0) at soo_write+0x24
writev(cd10f8a0,d29b8f80,28cf4e4c,28cf0280,0) at writev+0x188
syscall2(813002f,2f,bfbf002f,0,28cf0280) at syscall2+0x1f5
Xint0x80_syscall() at Xint0x80_syscall+0x25

The other one looks like this:

Fatal trap 12: page fault while in kernel mode
fault virtual address	= 0xc0cc25d4
fault code		= supervisor write, page not present
instruction pointer	= 0x8:0xc0201da3
stack pointer	        = 0x10:0xc02b7004
frame pointer	        = 0x10:0xc02b7010
code segment		= base 0x0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, def32 1, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= Idle
interrupt mask		= net tty 
kernel: type 12 trap, code=0
Stopped at      xl_newbuf+0x103:        incb    0(%eax,%edx,1)
db> tr
xl_newbuf(c13ae000,c13ae344) at xl_newbuf+0x103
xl_rxeof(c13ae000) at xl_rxeof+0x7e
xl_intr(c13ae000,660200,0,0,c02b70b8) at xl_intr+0x7c
intr_mux(c0b67920,0,10,10,400010) at intr_mux+0x1d
Xresume9() at Xresume9+0x2b
--- interrupt, eip = 0xc0262dd2, esp = 0xc02b70b0, ebp = 0xc02b70b8 ---
cpu_idle(e,634,2,80f9ff,0) at cpu_idle+0xe
idle_loop() at idle_loop+0x1d

Unfortunately I'm not in a position to remotely kgdb this, and swap is too
small for a dump.  I can tell you that in my kernel, sosend+0x443 is line
567 of kern/uipc_socket.c (revision 1.68.2.22), and xl_newbuf+0x103 is line
1817 of pci/if_xl.c (revision 1.72.2.25).

Both lines are MCLGET()s.

GENERIC also crashes on this machine.  I assume it's the same bug.

This might be related to some of the following:

http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/30952
http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/42260
http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/53717
http://lists.freebsd.org/pipermail/freebsd-current/2003-May/002215.html

Could this be an xl bug, in the card and/or in the FreeBSD driver,
triggered by the amount of memory in the machine?  Maybe software or
hardware attempts to DMA to high physical memory are going wrong?

>How-To-Repeat:
	Can't be easily reproduced, seems to happen randomly.
>Fix:

>Release-Note:
>Audit-Trail:
>Unformatted: