5.4-RC1 Freezing, but pingable (may be related to gvinum)

Sat Apr 9 12:48:06 PDT 2005

I wasn't sure if I should send this to -stable or -questions, so I 
apologize if I chose the wrong list.

I've had a pre-production server 'hang' twice  within the past couple of 
weeks. The first time the machine locked up it was running 5.4-BETA1 and 
the second time it was running 5.4-RC1. The machine ran fine and did not 
exhibit similar symptoms while running -STABLE for several months prior 
to cvsupping during the 5.4 cycle.

By "hang" I mean that the machine stops responding to console keystrokes 
(serial or otherwise) while existing ssh and nfs connections stop 
responding, but do not immediately close. The machine continues to 
respond to ICMP pings. I can not make new SSH connections and my 
attempts eventually timeout rather than giving me a connection refused 
error.

I wasn't able to find any clues in /var/log/messages.

The only obvious similarity that I've found with the two crashes is that 
the serial console cable had been accidentally unplugged at some point 
before the two crashes (clarification in the off chance that it matters: 
It's an RJ-45 from a portmaster to a RJ-45/DB9 adapter. The RJ-45 end 
was unplugged, while the adapter was still attached to the server.).

I tried to rule out the unpluggedness of the console cable contributing 
to this problem, by removing the console cable and letting the following 
run for a few minutes: # while true; do echo "blah" > /dev/console;done

This above loop ran for several minutes after unplugging the console 
cable without any obvious ill effects. I tried to plug/unplug the cable 
several times, while the loop ran.

After cvsupping to 5.4-Beta1 I enabled gvinum to use as a volume manager 
(RAID is being handled by a 3ware Escalade 9500-8).

I'd appreciate any help in troubleshooting this. I will copy/paste my 
gvinum configuration followed by dmesg.boot at the end of this e-mail.

Thanks,
-Ash

GVINUM:

gvinum -> printconfig
# Vinum configuration of golem.anhedonia.com, saved at Sat Apr  9 
14:36:20 2005
drive array00 device /dev/da1a
volume qbvol00
volume datavol00
plex name qbvol00.p0 org concat vol qbvol00
plex name datavol00.p0 org concat vol datavol00
sd name qbvol00.p0.s0 drive array00 len 20971520s driveoffset 265s plex 
qbvol00.p0 plexoffset 0s
sd name datavol00.p0.s0 drive array00 len 209715200s driveoffset 
20971785s plex datavol00.p0 plexoffset 0s
gvinum -> l
1 drive:
D array00               State: up       /dev/da1a       A: 840991/953631 
MB (88%)

2 volumes:
V qbvol00               State: up       Plexes:       1 Size:         10 GB
V datavol00             State: up       Plexes:       1 Size:        100 GB

2 plexes:
P qbvol00.p0          C State: up       Subdisks:     1 Size:         10 GB
P datavol00.p0        C State: up       Subdisks:     1 Size:        100 GB

2 subdisks:
S qbvol00.p0.s0         State: up       D: array00      Size:         10 GB
S datavol00.p0.s0       State: up       D: array00      Size:        100 GB

DMESG:

Copyright (c) 1992-2005 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
         The Regents of the University of California. All rights reserved.
FreeBSD 5.4-RC1 #0: Tue Apr  5 18:38:44 CDT 2005
     root at golem.anhedonia.com:/usr/obj/usr/src/sys/GOLEM
ACPI APIC Table: <PTLTD          APIC  >
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(TM) CPU 2.66GHz (2657.82-MHz 686-class CPU)
   Origin = "GenuineIntel"  Id = 0xf25  Stepping = 5

Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,C
MOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
   Hyperthreading: 2 logical CPUs
real memory  = 2146959360 (2047 MB)
avail memory = 2095423488 (1998 MB)
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
ioapic2 <Version 2.0> irqs 48-71 on motherboard
npx0: <math processor> on motherboard
npx0: INT 16 interface
acpi0: <PTLTD   RSDT> on motherboard
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
cpu0: <ACPI CPU> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pci0: <unknown> at device 0.1 (no driver attached)
pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pci1: <base peripheral, interrupt controller> at device 28.0 (no driver 
attached
)
pcib2: <ACPI PCI-PCI bridge> at device 29.0 on pci1
pci2: <ACPI PCI bus> on pcib2
pcib3: <PCI-PCI bridge> at device 2.0 on pci2
pci3: <PCI bus> on pcib3
em0: <Intel(R) PRO/1000 Network Connection, Version - 1.7.35> port 
0x7000-0x703f
  mem 0xfb200000-0xfb21ffff irq 28 at device 4.0 on pci3
em0: Ethernet address: 00:04:23:ad:73:e8
em0:  Speed:N/A  Duplex:N/A
em1: <Intel(R) PRO/1000 Network Connection, Version - 1.7.35> port 
0x7040-0x707f
  mem 0xfb220000-0xfb23ffff irq 29 at device 4.1 on pci3
em1: Ethernet address: 00:04:23:ad:73:e9
em1:  Speed:N/A  Duplex:N/A
em2: <Intel(R) PRO/1000 Network Connection, Version - 1.7.35> port 
0x7080-0x70bf
  mem 0xfb240000-0xfb25ffff irq 30 at device 6.0 on pci3
em2: Ethernet address: 00:04:23:ad:73:ea
em2:  Speed:N/A  Duplex:N/A
em3: <Intel(R) PRO/1000 Network Connection, Version - 1.7.35> port 
0x70c0-0x70ff
  mem 0xfb260000-0xfb27ffff irq 31 at device 6.1 on pci3
em3: Ethernet address: 00:04:23:ad:73:eb
em3:  Speed:N/A  Duplex:N/A
pci1: <base peripheral, interrupt controller> at device 30.0 (no driver 
attached
)
pcib4: <ACPI PCI-PCI bridge> at device 31.0 on pci1
pci4: <ACPI PCI bus> on pcib4
3ware device driver for 9000 series storage controllers, version: 
2.50.02.012
twa0: <3ware 9000 series Storage Controller> port 0x8000-0x80ff mem 
0xfd800000-0
xfdffffff,0xfb300000-0xfb3000ff irq 48 at device 1.0 on pci4
twa0: 8 ports, Firmware FE9X 2.04.00.005, BIOS BE9X 2.03.01.047
pci0: <unknown> at device 2.1 (no driver attached)
pcib5: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci5: <ACPI PCI bus> on pcib5
pci5: <display, VGA> at device 3.0 (no driver attached)
fxp0: <Intel 82550 Pro/100 Ethernet> port 0x9400-0x943f mem 
0xfb400000-0xfb41fff
f,0xfb441000-0xfb441fff irq 20 at device 4.0 on pci5
miibus0: <MII bus> on fxp0
inphy0: <i82555 10/100 media interface> on miibus0
inphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
fxp0: Ethernet address: 00:0e:0c:4e:02:5e
em4: <Intel(R) PRO/1000 Network Connection, Version - 1.7.35> port 
0x9440-0x947f
  mem 0xfb420000-0xfb43ffff irq 23 at device 5.0 on pci5
em4: Ethernet address: 00:0e:0c:4e:01:03
em4:  Speed:N/A  Duplex:N/A
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel ICH3 UDMA100 controller> port 
0x6c60-0x6c6f,0x376,0x170-0x177,0x
3f6,0x1f0-0x1f7 at device 31.1 on pci0
ata0: channel #0 on atapci0
ata1: channel #1 on atapci0
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
acpi_button0: <Power Button> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x64,0x60 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on 
acpi0
sio0: type 16550A, console
sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0
sio1: type 16550A
ppc0: <ECP parallel printer port> port 0x778-0x77b,0x378-0x37f irq 7 drq 
3 on ac
pi0
ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode
ppc0: FIFO with 16/16/9 bytes threshold
ppbus0: <Parallel port bus> on ppc0
plip0: <PLIP network interface> on ppbus0
lpt0: <Printer> on ppbus0
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
orm0: <ISA Option ROMs> at iomem 
0xe3000-0xe3fff,0xc8000-0xc97ff,0xc0000-0xc7fff
  on isa0
pmtimer0 on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounter "TSC" frequency 2657824944 Hz quality 800
Timecounters tick every 10.000 msec
acd0: CDROM <SR244W/T01A> at ata0-master UDMA33
da0 at twa0 bus 0 target 0 lun 0
da0: <3ware Logical Disk 00 1.00> Fixed Direct Access SCSI-0 device
da0: 100.000MB/s transfers
da0: 76283MB (156227584 512 byte sectors: 255H 63S/T 9724C)
da1 at twa0 bus 0 target 1 lun 0
da1: <3ware Logical Disk 01 1.00> Fixed Direct Access SCSI-0 device
da1: 100.000MB/s transfers
da1: 953632MB (1953038336 512 byte sectors: 255H 63S/T 121571C)
Mounting root from ufs:/dev/da0s1a