kern/69607: system crashes in if_tap module

[Josef Pojsl]

>Number:         69607
>Category:       kern
>Synopsis:       system crashes in if_tap module
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon Jul 26 08:00:41 GMT 2004
>Closed-Date:
>Last-Modified:
>Originator:     Josef Pojsl
>Release:        FreeBSD 4.10-RELEASE-p2 i386
>Organization:
Trusted Network Solutions, a.s., Czech rep.
>Environment:
uname -a:
FreeBSD  4.10-RELEASE-p2 FreeBSD 4.10-RELEASE-p2 #2: Thu Jul 22 13:28:42 CEST 2004     root@:/usr/src/sys/compile/GENERIC  i386

dmesg.boot:
Copyright (c) 1992-2004 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD 4.10-RELEASE-p2 #2: Thu Jul 22 13:28:42 CEST 2004
    root@:/usr/src/sys/compile/GENERIC
Timecounter "i8254"  frequency 1193182 Hz
CPU: Intel(R) Xeon(TM) CPU 2.40GHz (2392.29-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0xf25  Stepping = 5
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Hyperthreading: 2 logical CPUs
real memory  = 2147418112 (2097088K bytes)
avail memory = 2086227968 (2037332K bytes)
Warning: Pentium 4 CPU: PSE disabled
Pentium Pro MTRR support enabled
md0: Malloc disk
Using $PIR table, 19 entries at 0xc00f3460
npx0: <math processor> on motherboard
npx0: INT 16 interface
pcib0: <Host to PCI bridge> on motherboard
pci0: <PCI bus> on pcib0
pci0: <unknown card> (vendor=0x8086, dev=0x2541) at 0.1
pcib1: <PCI to PCI bridge (vendor=8086 device=2545)> at device 3.0 on pci0
pci2: <PCI bus> on pcib1
pci2: <unknown card> (vendor=0x8086, dev=0x1461) at 28.0
pcib2: <PCI to PCI bridge (vendor=8086 device=1460)> at device 29.0 on pci2
pci4: <PCI bus> on pcib2
pcib3: <PCI to PCI bridge (vendor=1014 device=01a7)> at device 2.0 on pci4
pci5: <PCI bus> on pcib3
em0: <Intel(R) PRO/1000 Network Connection, Version - 1.7.25> port 0x3040-0x307f mem 0xfeac0000-0xfeadffff irq 9 at device 4.0 on pci5
em0:  Speed:N/A  Duplex:N/A
em1: <Intel(R) PRO/1000 Network Connection, Version - 1.7.25> port 0x3000-0x303f mem 0xfeae0000-0xfeafffff irq 9 at device 4.1 on pci5
em1:  Speed:N/A  Duplex:N/A
em2: <Intel(R) PRO/1000 Network Connection, Version - 1.7.25> port 0x30c0-0x30ff mem 0xfea80000-0xfea9ffff irq 9 at device 6.0 on pci5
em2:  Speed:N/A  Duplex:N/A
em3: <Intel(R) PRO/1000 Network Connection, Version - 1.7.25> port 0x3080-0x30bf mem 0xfeaa0000-0xfeabffff irq 9 at device 6.1 on pci5
em3:  Speed:N/A  Duplex:N/A
pci2: <unknown card> (vendor=0x8086, dev=0x1461) at 30.0
pcib4: <PCI to PCI bridge (vendor=8086 device=1460)> at device 31.0 on pci2
pci3: <PCI bus> on pcib4
ahd0: <Adaptec AIC7901A Ultra320 SCSI adapter> port 0x2400-0x24ff,0x2000-0x20ff mem 0xfe9e0000-0xfe9e1fff irq 9 at device 3.0 on pci3
aic7901A: Ultra320 Wide Channel A, SCSI Id=7, PCI-X 67-100Mhz, 512 SCBs
pci0: <unknown card> (vendor=0x8086, dev=0x2546) at 3.1
uhci0: <Intel 82801CA/CAM (ICH3) USB controller USB-A> port 0x4040-0x405f irq 11 at device 29.0 on pci0
usb0: <Intel 82801CA/CAM (ICH3) USB controller USB-A> on uhci0
usb0: USB revision 1.0
uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
uhci1: <Intel 82801CA/CAM (ICH3) USB controller USB-B> port 0x4020-0x403f irq 5 at device 29.1 on pci0
usb1: <Intel 82801CA/CAM (ICH3) USB controller USB-B> on uhci1
usb1: USB revision 1.0
uhub1: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
uhci2: <Intel 82801CA/CAM (ICH3) USB controller USB-C> port 0x4000-0x401f irq 10 at device 29.2 on pci0
usb2: <Intel 82801CA/CAM (ICH3) USB controller USB-C> on uhci2
usb2: USB revision 1.0
uhub2: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub2: 2 ports with 2 removable, self powered
pcib5: <Intel 82801BA/BAM (ICH2) Hub to PCI bridge> at device 30.0 on pci0
pci1: <PCI bus> on pcib5
fxp0: <Intel 82550 Pro/100 Ethernet> port 0x1400-0x143f mem 0xfe4a0000-0xfe4bffff,0xfe4e0000-0xfe4e0fff irq 11 at device 3.0 on pci1
fxp0: Ethernet address 00:03:47:32:85:2f
inphy0: <i82555 10/100 media interface> on miibus0
inphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
em4: <Intel(R) PRO/1000 Network Connection, Version - 1.7.25> port 0x1440-0x147f mem 0xfe460000-0xfe47ffff irq 11 at device 4.0 on pci1
em4:  Speed:N/A  Duplex:N/A
pci1: <ATI Mach64-GR graphics accelerator> at 12.0 irq 11
isab0: <PCI to ISA bridge (vendor=8086 device=2480)> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel ICH3 ATA100 controller> port 0x3a0-0x3af,0-0x3,0-0x7,0-0x3,0-0x7 irq 0 at device 31.1 on pci0
ata0: at 0x1f0 irq 14 on atapci0
ata1: at 0x170 irq 15 on atapci0
pci0: <unknown card> (vendor=0x8086, dev=0x2483) at 31.3 irq 0
orm0: <Option ROMs> at iomem 0xc0000-0xc7fff,0xd2000-0xd37ff,0xd3800-0xd4fff on isa0
pmtimer0 on isa0
fdc0: <NEC 72065B or clone> at port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on isa0
fdc0: FIFO enabled, 8 bytes threshold
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> flags 0x1 irq 1 on atkbdc0
kbd0 at atkbd0
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
sio0: type 16550A
sio1 at port 0x2f8-0x2ff irq 3 on isa0
sio1: type 16550A
ppc0: <Parallel port> at port 0x378-0x37f irq 7 on isa0
ppc0: Generic chipset (ECP/PS2/NIBBLE) in COMPATIBLE mode
ppc0: FIFO with 16/16/8 bytes threshold
plip0: <PLIP network interface> on ppbus0
lpt0: <Printer> on ppbus0
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
acd0: MODE_SENSE_BIG command timeout - resetting
ata0: resetting devices .. done
acd0: CDROM <TEAC CD-552E> at ata0-master PIO4
Waiting 15 seconds for SCSI devices to settle
em3: Link is up 100 Mbps Full Duplex
Mounting root from ufs:da0s1a
da0 at ahd0 bus 0 target 8 lun 0
da0: <IBM DNES-309170W SA30> Fixed Direct Access SCSI-3 device 
da0: 40.000MB/s transfers (20.000MHz, offset 31, 16bit), Tagged Queueing Enabled
da0: 8748MB (17916240 512 byte sectors: 255H 63S/T 1115C)

>Description:
    - While using OpenVPN 1.6.0 (/usr/ports/security/openvpn),
    I have observed multiple system crashes after having loaded
    module if_tap AND having done ifconfig on the tap device,
    OR while kldunloading module if_tap.
    Mostly, it ends up with system dump, but sometimes system simply hangs.
    When dumping, system usually reports that the actual process
    is "ifconfig" and the reason for dump is "page fault",
    in seldom cases the process is "kldunload" and the reason is
    "general protection fault".

    - I have got the vmcore file but do not know what exactly to do with it.
    It has got 2GB, so I am not about to send it away. I tried
    running gdb -k and did a "bt" command (this is the first case,
    while doing ifconfig, the reason is page fault):
(kgdb) bt
#0  0xc022faa2 in dumpsys ()
#1  0xc022f873 in boot ()
#2  0xc022fc98 in poweroff_wait ()
#3  0xc03a7a6a in trap_fatal ()
#4  0xc03a773d in trap_pfault ()
#5  0xc03a72fb in trap ()
#6  0xc02831c1 in in_control ()
#7  0xc0273f82 in ifioctl ()
#8  0xc024224a in soo_ioctl ()
#9  0xc023f146 in ioctl ()
#10 0xc03a7d19 in syscall2 ()
#11 0xc0398d35 in Xint0x80_syscall ()
#12 0x80487b1 in ?? ()
#13 0x804813e in ?? ()
(kgdb)

    In another case, when the system crashes while unloading if_tap module
    (the reason is general protection fault now), the stack shows:
(kgdb) bt
#0  0xc022faa2 in dumpsys ()
#1  0xc022f873 in boot ()
#2  0xc022fc98 in poweroff_wait ()
#3  0xc03a7a6a in trap_fatal ()
#4  0xc03a7447 in trap ()
#5  0xc028af2e in rip_ctlinput ()
#6  0xc024ac76 in pfctlinput ()
#7  0xc02736b7 in if_unroute ()
#8  0xc027374b in if_down ()
#9  0xc0272e5a in if_detach ()
#10 0xc02755f8 in ether_ifdetach ()
#11 0xc4e89291 in ?? ()
#12 0xc021eee1 in module_unload ()
#13 0xc021f736 in linker_file_unload ()
#14 0xc021fbc9 in kldunload ()
#15 0xc03a7d19 in syscall2 ()
#16 0xc0398d35 in Xint0x80_syscall ()
#17 0x804813e in ?? ()
(kgdb)
    I am willing to provide further output if needed.

    - The problem does not take place always, only sometimes.
    One has to load if_tap, ifconfig and unload if_tap module several times
    before kernel dumps or hangs (see section How-To-Repeat below).

    - I have observed the same behavior on the following FreeBSD versions:
    4.9-RELEASE-p4, 4.9-RELEASE-p11, 4.10-RELEASE-p2

    - It seems that the problem disappears after having loaded if_tap
    by kernel (by including "pseudo-device tap" in kernel conf. file).
    Or at least, I have not been able to repeat the symptoms with that
    configuration.

>How-To-Repeat:
    1. Boot GENERIC kernel.
    2. Create the file crash4.c with the following contents:
#define TAP0 "/dev/tap0"
#define IFCONFIG_CMD "/sbin/ifconfig tap0 192.168.194.0 netmask 255.255.255.252 mtu 1500 up"

#include <stdlib.h>
#include <stdio.h>
#include <fcntl.h>

main() {

    if ( open ( TAP0, O_RDWR ) < 0 ) {
        fprintf( stderr, "Couldn't open %s\n", TAP0 );
        exit ( -1 );
    }

    if ( system ( IFCONFIG_CMD ) ) {
        fprintf( stderr, "System() failed\n" );
        exit ( -1 );
    }
}
    3. Compile crash4.c:
       bash# gcc crash4.c -o crash4
    4. Run the following command:
       bash# while true ; do kldload if_tap ; echo -n " loaded" ; sleep 1 ; ./crash4 ; echo -n " opened" ; sleep 1 ; kldunload if_tap ; echo -n " unloaded" ; sleep 1 ; done
    5. Wait until system dumps (on my system, it takes something
    between 1 and 60 minutes)
    6. [For some reason, the impact of the ifconfig command in crash4.c
    is not visible (i.e., after having run ./crash4, "ifconfig tap0"
    does not show any IP address), but the system dumps from time to time,
    anyway. Also, I was not able to achieve crash with a primitive shell batch.]
>Fix:
>Release-Note:
>Audit-Trail:
>Unformatted: