Total crash in gdb!.. something is broken!.. Was: Re: FreeBSD with a Gigabyte GA-K8NSC?

Johan Ström johan at stromnet.org
Sun Sep 24 05:23:52 PDT 2006


Okay, I got some problems here now... I'm trying to get clamav's  
clamd to work.. Failes with an abort in libc:

It coredumps directly on start, trace: http://sial.org/pbot/19922
Truss output: www.stromnet.org/~johan/clamd.log

Okay...something seems to be f*cked in the nss/ldap stuff..

Anyway,, when running with gdb --args /usr/local/sbin/clamd --debug  
it works fine!... No coredump or anything, untill i decide to kill  
clamd..

kill <pidofclamd> breaks into gdb, and when I run continue to process  
the signal and let it die, the whole fcking box dies! Screen goes  
black and reboot.. No panic messages or anything...
I have reproduced this two times now on the box in the dmesg in  
earlier mail... Then i moved the disk to another box pretty similar,  
same chipset i thikn but not exactly same mobo.. tried the above  
commands, and bam exactly same problem.. screen just goes black and  
the box reboots... Dmesg from that box:

if i can get the crap up running. now the fs is broken or some  
shit.... get this on boot, after started a few services:

Starting jails:/usr: bad dir ino 32125198 at offset 512: mangled entry
panic: ufs_dirbad: bad dir
Uptime: 54s
GEOM_MIRROR: Device gm0: provider mirror/gm0 destroyed
GEOM_MIRROR: Device gm0 destroyed.
Cannot dump. No dump device defined.
Automatic reboot in 15 seconds...

Ok, now i rebooted to singeluser mode and enalbed the dumpdev in  
rc.conf... , then saw it continune booting and i chcked for the line  
saying kernel dumps on /dev/mirror/gm0s1b... it was there... and then  
it booted further and got by the place it crashed before, but a  
minute later when i try to login to crashes on the same inode... AND  
STILL!.. it says Cannot dump. No dump device defined... WTF??...  
brokeness brokeness..

Okay, after some fscking its back up. dmesg from second box which i  
can crash with clamd...:

Copyright (c) 1992-2006 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
         The Regents of the University of California. All rights  
reserved.
FreeBSD 6.1-RELEASE-p7 #0: Wed Sep 20 09:21:41 CEST 2006
     admin at elfi.stromnet.org:/usr/obj/usr/src/sys/ELFI
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: AMD Athlon(tm) 64 Processor 3200+ (2210.09-MHz K8-class CPU)
   Origin = "AuthenticAMD"  Id = 0xfc0  Stepping = 0
    
Features=0x78bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE, 
MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2>
   AMD Features=0xe0500800<SYSCALL,NX,MMX+,LM,3DNow+,3DNow>
real memory  = 1073676288 (1023 MB)
avail memory = 1024299008 (976 MB)
ACPI APIC Table: <Nvidia AWRDACPI>
ioapic0 <Version 1.1> irqs 0-23 on motherboard
kbd1 at kbdmux0
acpi0: <Nvidia AWRDACPI> on motherboard
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x4008-0x400b on acpi0
cpu0: <ACPI CPU> on acpi0
acpi_button0: <Power Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff,0xcf0-0xcf3 on acpi0
pci0: <ACPI PCI bus> on pcib0
agp0: <NVIDIA nForce3-250 AGP Controller> mem 0xf0000000-0xf7ffffff  
at device 0.0 on pci0
isab0: <PCI-ISA bridge> at device 1.0 on pci0
isa0: <ISA bus> on isab0
pci0: <serial bus, SMBus> at device 1.1 (no driver attached)
ohci0: <OHCI (generic) USB controller> mem 0xfe02f000-0xfe02ffff irq  
21 at device 2.0 on pci0
ohci0: [GIANT-LOCKED]
usb0: OHCI version 1.0, legacy support
usb0: <OHCI (generic) USB controller> on ohci0
usb0: USB revision 1.0
uhub0: nVidia OHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 4 ports with 4 removable, self powered
ohci1: <OHCI (generic) USB controller> mem 0xfe02e000-0xfe02efff irq  
22 at device 2.1 on pci0
ohci1: [GIANT-LOCKED]
usb1: OHCI version 1.0, legacy support
usb1: <OHCI (generic) USB controller> on ohci1
usb1: USB revision 1.0
uhub1: nVidia OHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 4 ports with 4 removable, self powered
ehci0: <NVIDIA nForce3 250 USB 2.0 controller> mem  
0xfe02d000-0xfe02d0ff irq 23 at device 2.2 on pci0
ehci0: [GIANT-LOCKED]
usb2: EHCI version 1.0
usb2: companion controllers, 4 ports each: usb0 usb1
usb2: <NVIDIA nForce3 250 USB 2.0 controller> on ehci0
usb2: USB revision 2.0
uhub2: nVidia EHCI root hub, class 9/0, rev 2.00/1.00, addr 1
uhub2: 8 ports with 8 removable, self powered
nve0: <NVIDIA nForce MCP7 Networking Adapter> port 0xf000-0xf007 mem  
0xfe02c000-0xfe02cfff irq 21 at device 5.0 on pci0
nve0: Ethernet address 00:11:09:c5:fc:9e
miibus0: <MII bus> on nve0
ukphy0: <Generic IEEE 802.3u media interface> on miibus0
ukphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT,  
1000baseT-FDX, auto
nve0: Ethernet address: 00:11:09:c5:fc:9e
pci0: <multimedia, audio> at device 6.0 (no driver attached)
atapci0: <nVidia nForce3 Pro UDMA133 controller> port  
0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xdc00-0xdc0f at device 8.0 on pci0
ata0: <ATA channel 0> on atapci0
ata1: <ATA channel 1> on atapci0
atapci1: <nVidia nForce3 Pro SATA150 controller> port  
0x9e0-0x9e7,0xbe0-0xbe3,0x960-0x967,0xb60-0xb63,0xc800-0xc80f, 
0xc400-0xc47f irq 23 at device 9.0 on pci0
ata2: <ATA channel 0> on atapci1
ata3: <ATA channel 1> on atapci1
atapci2: <nVidia nForce3 Pro SATA150 controller> port  
0x9f0-0x9f7,0xbf0-0xbf3,0x970-0x977,0xb70-0xb73,0xb000-0xb00f, 
0xac00-0xac7f irq 20 at device 10.0 on pci0
ata4: <ATA channel 0> on atapci2
ata5: <ATA channel 1> on atapci2
pcib1: <ACPI PCI-PCI bridge> at device 11.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pci1: <display, VGA> at device 0.0 (no driver attached)
pcib2: <ACPI PCI-PCI bridge> at device 14.0 on pci0
pci2: <ACPI PCI bus> on pcib2
pci2: <multimedia, audio> at device 8.0 (no driver attached)
xl0: <3Com 3c905B-TX Fast Etherlink XL> port 0x8800-0x887f mem  
0xfdfff000-0xfdfff07f irq 19 at device 9.0 on pci2
miibus1: <MII bus> on xl0
xlphy0: <3Com internal media interface> on miibus1
xlphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
xl0: Ethernet address: 00:10:5a:dc:5e:aa
fwohci0: <VIA Fire II (VT6306)> port 0x8400-0x847f mem  
0xfdffe000-0xfdffe7ff irq 19 at device 12.0 on pci2
fwohci0: OHCI version 1.0 (ROM=1)
fwohci0: No. of Isochronous channels is 4.
fwohci0: EUI64 00:10:dc:00:00:77:83:dc
fwohci0: Phy 1394a available S400, 3 ports.
fwohci0: Link S400, max_rec 2048 bytes.
firewire0: <IEEE1394(FireWire) bus> on fwohci0
fwe0: <Ethernet over FireWire> on firewire0
if_fwe0: Fake Ethernet address: 02:10:dc:77:83:dc
fwe0: Ethernet address: 02:10:dc:77:83:dc
fwe0: if_start running deferred for Giant
sbp0: <SBP-2/SCSI over FireWire> on firewire0
fwohci0: Initiate bus reset
fwohci0: node_id=0xc800ffc0, gen=1, CYCLEMASTER mode
firewire0: 1 nodes, maxhop <= 0, cable IRM = 0 (me)
firewire0: bus manager 0 (me)
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on  
acpi0
fdc0: [FAST]
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10  
on acpi0
sio0: type 16550A
sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0
sio1: type 16550A
ppc0: <Standard parallel printer port> port 0x378-0x37f,0x778-0x77b  
irq 7 on acpi0
ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
ppbus0: <Parallel port bus> on ppc0
ppbus0: IEEE1284 device found /NIBBLE/ECP
Probing for PnP devices on ppbus0:
ppbus0: <HEWLETT-PACKARD DESKJET 820C> SCP,VLINK
plip0: <PLIP network interface> on ppbus0
lpt0: <Printer> on ppbus0
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> flags 0x1 irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
orm0: <ISA Option ROMs> at iomem 0xc0000-0xce7ff,0xd0000-0xd17ff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on  
isa0
ums0: Microsoft Microsoft 5-Button Mouse with IntelliEye(TM), rev  
1.10/3.00, addr 2, iclass 3/1
ums0: 5 buttons and Z dir.
Timecounter "TSC" frequency 2210091714 Hz quality 800
Timecounters tick every 1.000 msec
module_register_init: MOD_LOAD (amr_linux, 0xffffffff80654f90, 0)  
error 6
ad0: 156334MB <Maxtor 6Y160P4 YAR41BW0> at ata0-master UDMA133
acd0: DVDR <NEC DVD RW ND-3500AG/2.16> at ata1-master UDMA33
ad4: 286187MB <Maxtor 7L300S0 BANC1G10> at ata2-master SATA150
GEOM_MIRROR: Device gm0 created (id=316220990).
GEOM_MIRROR: Device gm0: provider ad4 detected.
GEOM_MIRROR: Device gm0: provider ad4 activated.
GEOM_MIRROR: Device gm0: provider mirror/gm0 launched.
Trying to mount root from ufs:/dev/mirror/gm0s1a


So.. Wtf is the problem here?...

Hope someone can help me..
Thanks

On Sep 23, 2006, at 18:41 , Johan Ström wrote:

>
> On Sep 3, 2006, at 14:13 , Johan Ström wrote:
>
>> Hi
>>
>> I'm about to get a "new" server... In this case what I'm looking  
>> at is a Gigabyte GA-K8NSC mobo with nForce3 250Gb chipset, and a  
>> AMD 64 3200+ Venice S939.
>>
>> Does anyone have any experience with FreeBSD (6.1) and this mobo/ 
>> chipset? Does the network work? How good? SATA? Any stability/ 
>> performance issues?
>>
>> I did notice it was mentioned on http://www.freebsd.org/platforms/ 
>> amd64/motherboards.html on 5.4 with the only comment "Sound and  
>> USB untested.".. So.. anyone got more detailed experience than that?
>>
>> Thanks :)
>> --
>> Johan Ström
>> johan at stromnet.org
>>
>
> Hi again,
> I got the mobo now and everything I've tested seems to work fine,  
> network (Marvell Gigabit Ethernet) works perfect (altough just  
> using 100mbit, havent tested gig), and sata seems to work..  
> Somewhat... Thats part of why I post this..
>
> I got two disks plugged in currently, two pieces of ad4: 286187MB  
> <Maxtor 7L300S0 BANC1G10> at ata2-master SATA150 (ad4 and ad6) on  
> one SATA each... When I only access ad4 (the system disk) and dont  
> touch ad6 (the old system disk, moving some data form there now..  
> soon to be gmirrored with ad4) it works fine.
> But as soon as i start to transer data from ad6 to ad4 (or rather,  
> from ad4s1f to gm0s1f of which ad6 is provider), the system becomes  
> veeerrry slow... Its still usable but it takes several seconds  
> (sometimes as much as 10-20) to ie exectue a simple command like  
> ls, top, su...
> gstat reports speeds of around 30MB/s:
>
> dT: 0.501  flag_I 500000us  sizeof 288  i -1
> L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
>    17    395     12    200  577.2    383  49059    8.5   99.2| ad4
>    17    395     12    200  577.2    383  49059    8.5   99.2|  
> mirror/gm0
>     0      0      0      0    0.0      0      0    0.0    0.0| ad4s1
>    17    395     12    200  577.2    383  49059    8.5   99.2|  
> mirror/gm0s1
>     0    387    387  49570    1.1      0      0    0.0   43.0| ad6
>     3      2      2     32  583.1      0      0    0.0  116.5|  
> mirror/gm0s1a
>     1      0      0      0    0.0      0      0    0.0    0.0|  
> mirror/gm0s1b
>     0      0      0      0    0.0      0      0    0.0    0.0|  
> mirror/gm0s1c
>     0      0      0      0    0.0      0      0    0.0    0.0|  
> mirror/gm0s1d
>     0      0      0      0    0.0      0      0    0.0    0.0|  
> mirror/gm0s1e
>    13    393     10    168  576.0    383  49059    8.5   95.3|  
> mirror/gm0s1f
>     0    387    387  49570    1.1      0      0    0.0   43.2| ad6s1
>     0      0      0      0    0.0      0      0    0.0    0.0| ad6s1a
>     0      0      0      0    0.0      0      0    0.0    0.0| ad6s1b
>     0      0      0      0    0.0      0      0    0.0    0.0| ad6s1c
>     0      0      0      0    0.0      0      0    0.0    0.0| ad6s1d
>     0      0      0      0    0.0      0      0    0.0    0.0| ad6s1e
>     0    387    387  49570    1.1      0      0    0.0   44.0| ad6s1f
>
> Those busy figures.. on the gmirror they fly up to > 100% all the  
> time and are red.. on the ad6 figures they are 40-50% all the time  
> (during copy that is)..
>
> Any ideas?
>
> dmesg:
>
> Copyright (c) 1992-2006 The FreeBSD Project.
> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993,  
> 1994
>         The Regents of the University of California. All rights  
> reserved.
> FreeBSD 6.1-RELEASE-p7 #0: Wed Sep 20 09:21:41 CEST 2006
>     admin at elfi.stromnet.org:/usr/obj/usr/src/sys/ELFI
> Timecounter "i8254" frequency 1193182 Hz quality 0
> CPU: AMD Athlon(tm) 64 Processor 3200+ (2009.79-MHz K8-class CPU)
>   Origin = "AuthenticAMD"  Id = 0x20ff0  Stepping = 0
>    
> Features=0x78bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PG 
> E,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2>
>   Features2=0x1<SSE3>
>   AMD Features=0xe2500800<SYSCALL,NX,MMX+,FFXSR,LM,3DNow+,3DNow>
>   AMD Features2=0x1<LAHF>
> real memory  = 1073676288 (1023 MB)
> avail memory = 1024299008 (976 MB)
> ACPI APIC Table: <Nvidia AWRDACPI>
> ioapic0 <Version 1.1> irqs 0-23 on motherboard
> kbd1 at kbdmux0
> acpi0: <Nvidia AWRDACPI> on motherboard
> acpi0: Power Button (fixed)
> Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
> acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
> cpu0: <ACPI CPU> on acpi0
> acpi_button0: <Power Button> on acpi0
> pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff,0xcf0-0xcf3 on acpi0
> pci0: <ACPI PCI bus> on pcib0
> agp0: <NVIDIA nForce3-250 AGP Controller> mem 0xf8000000-0xf9ffffff  
> at device 0.0 on pci0
> isab0: <PCI-ISA bridge> at device 1.0 on pci0
> isa0: <ISA bus> on isab0
> pci0: <serial bus, SMBus> at device 1.1 (no driver attached)
> ohci0: <OHCI (generic) USB controller> mem 0xfd005000-0xfd005fff  
> irq 20 at device 2.0 on pci0
> ohci0: [GIANT-LOCKED]
> usb0: OHCI version 1.0, legacy support
> usb0: SMM does not respond, resetting
> usb0: <OHCI (generic) USB controller> on ohci0
> usb0: USB revision 1.0
> uhub0: nVidia OHCI root hub, class 9/0, rev 1.00/1.00, addr 1
> uhub0: 4 ports with 4 removable, self powered
> ohci1: <OHCI (generic) USB controller> mem 0xfd000000-0xfd000fff  
> irq 21 at device 2.1 on pci0
> ohci1: [GIANT-LOCKED]
> usb1: OHCI version 1.0, legacy support
> usb1: SMM does not respond, resetting
> usb1: <OHCI (generic) USB controller> on ohci1
> usb1: USB revision 1.0
> uhub1: nVidia OHCI root hub, class 9/0, rev 1.00/1.00, addr 1
> uhub1: 4 ports with 4 removable, self powered
> ehci0: <NVIDIA nForce3 250 USB 2.0 controller> mem  
> 0xfd001000-0xfd0010ff irq 22 at device 2.2 on pci0
> ehci0: [GIANT-LOCKED]
> usb2: EHCI version 1.0
> usb2: companion controllers, 4 ports each: usb0 usb1
> usb2: <NVIDIA nForce3 250 USB 2.0 controller> on ehci0
> usb2: USB revision 2.0
> uhub2: nVidia EHCI root hub, class 9/0, rev 2.00/1.00, addr 1
> uhub2: 8 ports with 8 removable, self powered
> pci0: <multimedia, audio> at device 6.0 (no driver attached)
> atapci0: <nVidia nForce3 Pro UDMA133 controller> port  
> 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xf000-0xf00f at device 8.0 on  
> pci0
> ata0: <ATA channel 0> on atapci0
> ata1: <ATA channel 1> on atapci0
> atapci1: <nVidia nForce3 Pro SATA150 controller> port  
> 0x9f0-0x9f7,0xbf0-0xbf3,0x970-0x977,0xb70-0xb73,0xdc00-0xdc0f, 
> 0xe000-0xe07f irq 21 at device 10.0 on pci0
> ata2: <ATA channel 0> on atapci1
> ata3: <ATA channel 1> on atapci1
> pcib1: <ACPI PCI-PCI bridge> at device 11.0 on pci0
> pci1: <ACPI PCI bus> on pcib1
> pcib2: <ACPI PCI-PCI bridge> at device 14.0 on pci0
> pci2: <ACPI PCI bus> on pcib2
> pci2: <display, VGA> at device 10.0 (no driver attached)
> skc0: <Marvell Gigabit Ethernet> port 0xa400-0xa4ff mem  
> 0xfc000000-0xfc003fff irq 19 at device 11.0 on pci2
> skc0: Marvell Yukon Lite Gigabit Ethernet rev. (0x9)
> sk0: <Marvell Semiconductor, Inc. Yukon> on skc0
> sk0: Ethernet address: 00:14:85:21:c4:44
> miibus0: <MII bus> on sk0
> e1000phy0: <Marvell 88E1000 Gigabit PHY> on miibus0
> e1000phy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX,  
> 1000baseTX-FDX, auto
> fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2  
> on acpi0
> fdc0: [FAST]
> sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags  
> 0x10 on acpi0
> sio0: type 16550A
> sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0
> sio1: type 16550A
> ppc0: <ECP parallel printer port> port 0x378-0x37f,0x778-0x77b irq  
> 7 drq 3 on acpi0
> ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode
> ppc0: FIFO with 16/16/16 bytes threshold
> ppbus0: <Parallel port bus> on ppc0
> plip0: <PLIP network interface> on ppbus0
> lpt0: <Printer> on ppbus0
> lpt0: Interrupt-driven port
> ppi0: <Parallel I/O> on ppbus0
> atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
> atkbd0: <AT Keyboard> flags 0x1 irq 1 on atkbdc0
> kbd0 at atkbd0
> atkbd0: [GIANT-LOCKED]
> orm0: <ISA Option ROM> at iomem 0xc0000-0xc7fff on isa0
> sc0: <System console> at flags 0x100 on isa0
> sc0: VGA <16 virtual consoles, flags=0x300>
> vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff  
> on isa0
> Timecounter "TSC" frequency 2009790446 Hz quality 800
> Timecounters tick every 1.000 msec
> module_register_init: MOD_LOAD (amr_linux, 0xffffffff80654f90, 0)  
> error 6
> ad4: 286187MB <Maxtor 7L300S0 BANC1G10> at ata2-master SATA150
>
> elfi# gmirror list
> Geom name: gm0
> State: COMPLETE
> Components: 1
> Balance: round-robin
> Slice: 4096
> Flags: NONE
> GenID: 0
> SyncID: 1
> ID: 316220990
> Providers:
> 1. Name: mirror/gm0
>    Mediasize: 300089646080 (279G)
>    Sectorsize: 512
>    Mode: r5w5e6
> Consumers:
> 1. Name: ad4
>    Mediasize: 300089646592 (279G)
>    Sectorsize: 512
>    Mode: r1w1e1
>    State: ACTIVE
>    Priority: 0
>    Flags: DIRTY
>    GenID: 0
>    SyncID: 1
>    ID: 3833124755
>
>
> I just saw http://www.freebsd.org/releases/6.1R/installation- 
> amd64.html says:
>
> If you have an machine based on an nVidia nForce3 Pro-150, you MUST  
> use the BIOS setup to disable the IO APIC. If you do not have an  
> option to do this, you will likely have to disable ACPI instead.  
> There are bugs in the Pro-150 chipset that we have not found a  
> workaround for yet.
>
> Could this affect? Might try that...
>
> Thanks
> Johan
>
>>
>>
>> _______________________________________________
>> freebsd-hardware at freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-hardware
>> To unsubscribe, send any mail to "freebsd-hardware- 
>> unsubscribe at freebsd.org"
>
> _______________________________________________
> freebsd-stable at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable- 
> unsubscribe at freebsd.org"



More information about the freebsd-stable mailing list