use after free panic ZFS
Larry Rosenman
ler at lerctr.org
Mon May 18 18:07:05 UTC 2015
I've filed: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=200288 for
this.
I'd appreciate someone looking at it.
I'm STUCK here.
On 2015-05-18 07:56, Larry Rosenman wrote:
> On Mon, May 18, 2015 at 07:42:47AM -0500, Larry Rosenman wrote:
>> found the following panic this am:
>>
>> borg.lerctr.org dumped core - see /var/crash/vmcore.5
>>
>> Sun May 17 23:47:48 CDT 2015
>>
>> FreeBSD borg.lerctr.org 11.0-CURRENT FreeBSD 11.0-CURRENT #40 r283007:
>> Sat May 16 07:23:43 CDT 2015
>> root at borg.lerctr.org:/usr/obj/usr/src/sys/VT-LER amd64
>>
>> panic: Most recently used by solaris
>>
>> GNU gdb 6.1.1 [FreeBSD]
>> Copyright 2004 Free Software Foundation, Inc.
>> GDB is free software, covered by the GNU General Public License, and
>> you are
>> welcome to change it and/or distribute copies of it under certain
>> conditions.
>> Type "show copying" to see the conditions.
>> There is absolutely no warranty for GDB. Type "show warranty" for
>> details.
>> This GDB was configured as "amd64-marcel-freebsd"...
>>
>> Unread portion of the kernel message buffer:
>> Memory modified after free 0xfffff808535ea000(120) val=deadc0dd @
>> 0xfffff808535ea050
>> panic: Most recently used by solaris
>>
>> cpuid = 5
>> KDB: stack backtrace:
>> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
>> 0xfffffe100bfb7660
>> vpanic() at vpanic+0x189/frame 0xfffffe100bfb76e0
>> panic() at panic+0x43/frame 0xfffffe100bfb7740
>> mtrash_dtor() at mtrash_dtor/frame 0xfffffe100bfb7760
>> uma_zalloc_arg() at uma_zalloc_arg+0x4c2/frame 0xfffffe100bfb77d0
>> malloc() at malloc+0x198/frame 0xfffffe100bfb7820
>> zfs_range_lock() at zfs_range_lock+0x4a/frame 0xfffffe100bfb7880
>> zfs_get_data() at zfs_get_data+0x14c/frame 0xfffffe100bfb78f0
>> zil_commit() at zil_commit+0x94c/frame 0xfffffe100bfb7a10
>> zfs_freebsd_fsync() at zfs_freebsd_fsync+0xc8/frame 0xfffffe100bfb7a40
>> VOP_FSYNC_APV() at VOP_FSYNC_APV+0xf7/frame 0xfffffe100bfb7a70
>> sys_fsync() at sys_fsync+0x173/frame 0xfffffe100bfb7ae0
>> amd64_syscall() at amd64_syscall+0x25a/frame 0xfffffe100bfb7bf0
>> Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe100bfb7bf0
>> --- syscall (95, FreeBSD ELF64, sys_fsync), rip = 0x801eb5daa, rsp =
>> 0x7fffffffd598, rbp = 0x7fffffffd5b0 ---
>> Uptime: 1d14h25m26s
>> Dumping 12469 out of 64457
>> MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%
>>
>> Reading symbols from /boot/kernel/linux.ko.symbols...done.
>> Loaded symbols for /boot/kernel/linux.ko.symbols
>> Reading symbols from /boot/kernel/if_lagg.ko.symbols...done.
>> Loaded symbols for /boot/kernel/if_lagg.ko.symbols
>> Reading symbols from /boot/kernel/snd_envy24ht.ko.symbols...done.
>> Loaded symbols for /boot/kernel/snd_envy24ht.ko.symbols
>> Reading symbols from /boot/kernel/snd_spicds.ko.symbols...done.
>> Loaded symbols for /boot/kernel/snd_spicds.ko.symbols
>> Reading symbols from /boot/kernel/coretemp.ko.symbols...done.
>> Loaded symbols for /boot/kernel/coretemp.ko.symbols
>> Reading symbols from /boot/kernel/ichsmb.ko.symbols...done.
>> Loaded symbols for /boot/kernel/ichsmb.ko.symbols
>> Reading symbols from /boot/kernel/smbus.ko.symbols...done.
>> Loaded symbols for /boot/kernel/smbus.ko.symbols
>> Reading symbols from /boot/kernel/ichwd.ko.symbols...done.
>> Loaded symbols for /boot/kernel/ichwd.ko.symbols
>> Reading symbols from /boot/kernel/cpuctl.ko.symbols...done.
>> Loaded symbols for /boot/kernel/cpuctl.ko.symbols
>> Reading symbols from /boot/kernel/crypto.ko.symbols...done.
>> Loaded symbols for /boot/kernel/crypto.ko.symbols
>> Reading symbols from /boot/kernel/cryptodev.ko.symbols...done.
>> Loaded symbols for /boot/kernel/cryptodev.ko.symbols
>> Reading symbols from /boot/kernel/dtraceall.ko.symbols...done.
>> Loaded symbols for /boot/kernel/dtraceall.ko.symbols
>> Reading symbols from /boot/kernel/profile.ko.symbols...done.
>> Loaded symbols for /boot/kernel/profile.ko.symbols
>> Reading symbols from /boot/kernel/dtrace.ko.symbols...done.
>> Loaded symbols for /boot/kernel/dtrace.ko.symbols
>> Reading symbols from
>> /boot/kernel/systrace_freebsd32.ko.symbols...done.
>> Loaded symbols for /boot/kernel/systrace_freebsd32.ko.symbols
>> Reading symbols from /boot/kernel/systrace.ko.symbols...done.
>> Loaded symbols for /boot/kernel/systrace.ko.symbols
>> Reading symbols from /boot/kernel/sdt.ko.symbols...done.
>> Loaded symbols for /boot/kernel/sdt.ko.symbols
>> Reading symbols from /boot/kernel/lockstat.ko.symbols...done.
>> Loaded symbols for /boot/kernel/lockstat.ko.symbols
>> Reading symbols from /boot/kernel/fasttrap.ko.symbols...done.
>> Loaded symbols for /boot/kernel/fasttrap.ko.symbols
>> Reading symbols from /boot/kernel/fbt.ko.symbols...done.
>> Loaded symbols for /boot/kernel/fbt.ko.symbols
>> Reading symbols from /boot/kernel/dtnfscl.ko.symbols...done.
>> Loaded symbols for /boot/kernel/dtnfscl.ko.symbols
>> Reading symbols from /boot/kernel/dtmalloc.ko.symbols...done.
>> Loaded symbols for /boot/kernel/dtmalloc.ko.symbols
>> Reading symbols from /boot/modules/vboxdrv.ko...done.
>> Loaded symbols for /boot/modules/vboxdrv.ko
>> Reading symbols from /boot/modules/nvidia.ko...done.
>> Loaded symbols for /boot/modules/nvidia.ko
>> Reading symbols from /boot/kernel/ipmi.ko.symbols...done.
>> Loaded symbols for /boot/kernel/ipmi.ko.symbols
>> Reading symbols from /boot/kernel/ipmi_linux.ko.symbols...done.
>> Loaded symbols for /boot/kernel/ipmi_linux.ko.symbols
>> Reading symbols from /boot/kernel/radeonkms.ko.symbols...done.
>> Loaded symbols for /boot/kernel/radeonkms.ko.symbols
>> Reading symbols from /boot/kernel/iicbb.ko.symbols...done.
>> Loaded symbols for /boot/kernel/iicbb.ko.symbols
>> Reading symbols from /boot/kernel/iicbus.ko.symbols...done.
>> Loaded symbols for /boot/kernel/iicbus.ko.symbols
>> Reading symbols from /boot/kernel/iic.ko.symbols...done.
>> Loaded symbols for /boot/kernel/iic.ko.symbols
>> Reading symbols from /boot/kernel/drm2.ko.symbols...done.
>> Loaded symbols for /boot/kernel/drm2.ko.symbols
>> Reading symbols from
>> /boot/kernel/radeonkmsfw_R100_cp.ko.symbols...done.
>> Loaded symbols for /boot/kernel/radeonkmsfw_R100_cp.ko.symbols
>> Reading symbols from /boot/kernel/uhid.ko.symbols...done.
>> Loaded symbols for /boot/kernel/uhid.ko.symbols
>> Reading symbols from /boot/kernel/ums.ko.symbols...done.
>> Loaded symbols for /boot/kernel/ums.ko.symbols
>> Reading symbols from /boot/modules/vboxnetflt.ko...done.
>> Loaded symbols for /boot/modules/vboxnetflt.ko
>> Reading symbols from /boot/kernel/netgraph.ko.symbols...done.
>> Loaded symbols for /boot/kernel/netgraph.ko.symbols
>> Reading symbols from /boot/kernel/ng_ether.ko.symbols...done.
>> Loaded symbols for /boot/kernel/ng_ether.ko.symbols
>> Reading symbols from /boot/modules/vboxnetadp.ko...done.
>> Loaded symbols for /boot/modules/vboxnetadp.ko
>> #0 doadump (textdump=Unhandled dwarf expression opcode 0x93
>> ) at pcpu.h:221
>> 221 pcpu.h: No such file or directory.
>> in pcpu.h
>> (kgdb) #0 doadump (textdump=Unhandled dwarf expression opcode 0x93
>> ) at pcpu.h:221
>> #1 0xffffffff80a839b5 in kern_reboot (howto=Unhandled dwarf
>> expression opcode 0x93
>> )
>> at /usr/src/sys/kern/kern_shutdown.c:447
>> #2 0xffffffff80a83fa8 in vpanic (fmt=<value optimized out>,
>> ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:744
>> #3 0xffffffff80a83ff3 in panic (fmt=0x0)
>> at /usr/src/sys/kern/kern_shutdown.c:675
>> #4 0xffffffff80d13750 in mtrash_ctor (mem=<value optimized out>,
>> size=<value optimized out>, arg=<value optimized out>,
>> flags=<value optimized out>) at /usr/src/sys/vm/uma_dbg.c:138
>> #5 0xffffffff80d0f6d2 in uma_zalloc_arg (zone=0xfffff80ffffc9680,
>> udata=0x0,
>> flags=2) at /usr/src/sys/vm/uma_core.c:2197
>> #6 0xffffffff80a64158 in malloc (size=<value optimized out>,
>> mtp=0xffffffff815e16e0, flags=<value optimized out>) at uma.h:336
>> #7 0xffffffff80402b4a in zfs_range_lock (zp=0xfffff8075e835730,
>> off=9158656,
>> len=8192, type=Unhandled dwarf expression opcode 0x93
>> )
>> at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_rlock.c:432
>> #8 0xffffffff8040886c in zfs_get_data (arg=<value optimized out>,
>> lr=<value optimized out>,
>> buf=0xfffffe0662be8178 <Address 0xfffffe0662be8178 out of bounds>,
>> zio=0xfffff80d78b89ac8)
>> at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:1250
>> #9 0xffffffff8041c71c in zil_commit (zilog=0xfffff800185c1400,
>> foid=<value optimized out>)
>> at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zil.c:1108
>> #10 0xffffffff80410168 in zfs_freebsd_fsync (ap=<value optimized out>)
>> at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:2747
>> #11 0xffffffff80fdfcd7 in VOP_FSYNC_APV (vop=<value optimized out>,
>> a=<value optimized out>) at vnode_if.c:1328
>> #12 0xffffffff80b40883 in sys_fsync (td=0xfffff8011b253940,
>> uap=<value optimized out>) at vnode_if.h:549
>> #13 0xffffffff80e968da in amd64_syscall (td=0xfffff8011b253940,
>> traced=0)
>> at subr_syscall.c:133
>> #14 0xffffffff80e767bb in Xfast_syscall ()
>> at /usr/src/sys/amd64/amd64/exception.S:395
>> #15 0x0000000801eb5daa in ?? ()
>> Previous frame inner to this frame (corrupt stack?)
>> Current language: auto; currently minimal
>> (kgdb)
>>
>> I have the core.
> And, trying to re-compile to pick up the latest, got another one:
>
> borg.lerctr.org dumped core - see /var/crash/vmcore.6
>
> Mon May 18 07:51:57 CDT 2015
>
> FreeBSD borg.lerctr.org 11.0-CURRENT FreeBSD 11.0-CURRENT #40 r283007:
> Sat May 16 07:23:43 CDT 2015
> root at borg.lerctr.org:/usr/obj/usr/src/sys/VT-LER amd64
>
> panic: Most recently used by solaris
>
> GNU gdb 6.1.1 [FreeBSD]
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and
> you are
> welcome to change it and/or distribute copies of it under certain
> conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB. Type "show warranty" for
> details.
> This GDB was configured as "amd64-marcel-freebsd"...
>
> Unread portion of the kernel message buffer:
> Memory modified after free 0xfffff80c88f1f980(120) val=deadc0dd @
> 0xfffff80c88f1f9c0
> panic: Most recently used by solaris
>
> cpuid = 0
> KDB: stack backtrace:
> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
> 0xfffffe100c459600
> vpanic() at vpanic+0x189/frame 0xfffffe100c459680
> panic() at panic+0x43/frame 0xfffffe100c4596e0
> mtrash_dtor() at mtrash_dtor/frame 0xfffffe100c459700
> uma_zalloc_arg() at uma_zalloc_arg+0x4c2/frame 0xfffffe100c459770
> malloc() at malloc+0x198/frame 0xfffffe100c4597c0
> zfs_range_lock() at zfs_range_lock+0x4a/frame 0xfffffe100c459820
> zfs_freebsd_read() at zfs_freebsd_read+0x1c7/frame 0xfffffe100c4598c0
> VOP_READ_APV() at VOP_READ_APV+0xf1/frame 0xfffffe100c4598f0
> vn_read() at vn_read+0x237/frame 0xfffffe100c459970
> vn_io_fault() at vn_io_fault+0x10a/frame 0xfffffe100c4599f0
> dofileread() at dofileread+0x95/frame 0xfffffe100c459a40
> kern_readv() at kern_readv+0x68/frame 0xfffffe100c459a90
> sys_read() at sys_read+0x63/frame 0xfffffe100c459ae0
> amd64_syscall() at amd64_syscall+0x25a/frame 0xfffffe100c459bf0
> Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe100c459bf0
> --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x8009638fa, rsp =
> 0x7fffffffe968, rbp = 0x7fffffffe980 ---
> Uptime: 7h59m25s
> Dumping 14815 out of 64457
> MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%
>
> Reading symbols from /boot/kernel/linux.ko.symbols...done.
> Loaded symbols for /boot/kernel/linux.ko.symbols
> Reading symbols from /boot/kernel/if_lagg.ko.symbols...done.
> Loaded symbols for /boot/kernel/if_lagg.ko.symbols
> Reading symbols from /boot/kernel/snd_envy24ht.ko.symbols...done.
> Loaded symbols for /boot/kernel/snd_envy24ht.ko.symbols
> Reading symbols from /boot/kernel/snd_spicds.ko.symbols...done.
> Loaded symbols for /boot/kernel/snd_spicds.ko.symbols
> Reading symbols from /boot/kernel/coretemp.ko.symbols...done.
> Loaded symbols for /boot/kernel/coretemp.ko.symbols
> Reading symbols from /boot/kernel/ichsmb.ko.symbols...done.
> Loaded symbols for /boot/kernel/ichsmb.ko.symbols
> Reading symbols from /boot/kernel/smbus.ko.symbols...done.
> Loaded symbols for /boot/kernel/smbus.ko.symbols
> Reading symbols from /boot/kernel/ichwd.ko.symbols...done.
> Loaded symbols for /boot/kernel/ichwd.ko.symbols
> Reading symbols from /boot/kernel/cpuctl.ko.symbols...done.
> Loaded symbols for /boot/kernel/cpuctl.ko.symbols
> Reading symbols from /boot/kernel/crypto.ko.symbols...done.
> Loaded symbols for /boot/kernel/crypto.ko.symbols
> Reading symbols from /boot/kernel/cryptodev.ko.symbols...done.
> Loaded symbols for /boot/kernel/cryptodev.ko.symbols
> Reading symbols from /boot/kernel/dtraceall.ko.symbols...done.
> Loaded symbols for /boot/kernel/dtraceall.ko.symbols
> Reading symbols from /boot/kernel/profile.ko.symbols...done.
> Loaded symbols for /boot/kernel/profile.ko.symbols
> Reading symbols from /boot/kernel/dtrace.ko.symbols...done.
> Loaded symbols for /boot/kernel/dtrace.ko.symbols
> Reading symbols from /boot/kernel/systrace_freebsd32.ko.symbols...done.
> Loaded symbols for /boot/kernel/systrace_freebsd32.ko.symbols
> Reading symbols from /boot/kernel/systrace.ko.symbols...done.
> Loaded symbols for /boot/kernel/systrace.ko.symbols
> Reading symbols from /boot/kernel/sdt.ko.symbols...done.
> Loaded symbols for /boot/kernel/sdt.ko.symbols
> Reading symbols from /boot/kernel/lockstat.ko.symbols...done.
> Loaded symbols for /boot/kernel/lockstat.ko.symbols
> Reading symbols from /boot/kernel/fasttrap.ko.symbols...done.
> Loaded symbols for /boot/kernel/fasttrap.ko.symbols
> Reading symbols from /boot/kernel/fbt.ko.symbols...done.
> Loaded symbols for /boot/kernel/fbt.ko.symbols
> Reading symbols from /boot/kernel/dtnfscl.ko.symbols...done.
> Loaded symbols for /boot/kernel/dtnfscl.ko.symbols
> Reading symbols from /boot/kernel/dtmalloc.ko.symbols...done.
> Loaded symbols for /boot/kernel/dtmalloc.ko.symbols
> Reading symbols from /boot/modules/vboxdrv.ko...done.
> Loaded symbols for /boot/modules/vboxdrv.ko
> Reading symbols from /boot/modules/nvidia.ko...done.
> Loaded symbols for /boot/modules/nvidia.ko
> Reading symbols from /boot/kernel/ipmi.ko.symbols...done.
> Loaded symbols for /boot/kernel/ipmi.ko.symbols
> Reading symbols from /boot/kernel/ipmi_linux.ko.symbols...done.
> Loaded symbols for /boot/kernel/ipmi_linux.ko.symbols
> Reading symbols from /boot/kernel/radeonkms.ko.symbols...done.
> Loaded symbols for /boot/kernel/radeonkms.ko.symbols
> Reading symbols from /boot/kernel/iicbb.ko.symbols...done.
> Loaded symbols for /boot/kernel/iicbb.ko.symbols
> Reading symbols from /boot/kernel/iicbus.ko.symbols...done.
> Loaded symbols for /boot/kernel/iicbus.ko.symbols
> Reading symbols from /boot/kernel/iic.ko.symbols...done.
> Loaded symbols for /boot/kernel/iic.ko.symbols
> Reading symbols from /boot/kernel/drm2.ko.symbols...done.
> Loaded symbols for /boot/kernel/drm2.ko.symbols
> Reading symbols from
> /boot/kernel/radeonkmsfw_R100_cp.ko.symbols...done.
> Loaded symbols for /boot/kernel/radeonkmsfw_R100_cp.ko.symbols
> Reading symbols from /boot/kernel/uhid.ko.symbols...done.
> Loaded symbols for /boot/kernel/uhid.ko.symbols
> Reading symbols from /boot/kernel/ums.ko.symbols...done.
> Loaded symbols for /boot/kernel/ums.ko.symbols
> Reading symbols from /boot/modules/vboxnetflt.ko...done.
> Loaded symbols for /boot/modules/vboxnetflt.ko
> Reading symbols from /boot/kernel/netgraph.ko.symbols...done.
> Loaded symbols for /boot/kernel/netgraph.ko.symbols
> Reading symbols from /boot/kernel/ng_ether.ko.symbols...done.
> Loaded symbols for /boot/kernel/ng_ether.ko.symbols
> Reading symbols from /boot/modules/vboxnetadp.ko...done.
> Loaded symbols for /boot/modules/vboxnetadp.ko
> #0 doadump (textdump=Unhandled dwarf expression opcode 0x93
> ) at pcpu.h:221
> 221 pcpu.h: No such file or directory.
> in pcpu.h
> (kgdb) #0 doadump (textdump=Unhandled dwarf expression opcode 0x93
> ) at pcpu.h:221
> #1 0xffffffff80a839b5 in kern_reboot (howto=Unhandled dwarf
> expression opcode 0x93
> )
> at /usr/src/sys/kern/kern_shutdown.c:447
> #2 0xffffffff80a83fa8 in vpanic (fmt=<value optimized out>,
> ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:744
> #3 0xffffffff80a83ff3 in panic (fmt=0x0)
> at /usr/src/sys/kern/kern_shutdown.c:675
> #4 0xffffffff80d13750 in mtrash_ctor (mem=<value optimized out>,
> size=<value optimized out>, arg=<value optimized out>,
> flags=<value optimized out>) at /usr/src/sys/vm/uma_dbg.c:138
> #5 0xffffffff80d0f6d2 in uma_zalloc_arg (zone=0xfffff80ffffc9680,
> udata=0x0,
> flags=2) at /usr/src/sys/vm/uma_core.c:2197
> #6 0xffffffff80a64158 in malloc (size=<value optimized out>,
> mtp=0xffffffff815e16e0, flags=<value optimized out>) at uma.h:336
> #7 0xffffffff80402b4a in zfs_range_lock (zp=0xfffff806afc8d170,
> off=75316383, len=131072, type=Unhandled dwarf expression opcode
> 0x93
> )
> at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_rlock.c:432
> #8 0xffffffff8040e517 in zfs_freebsd_read (ap=<value optimized out>)
> at
> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:703
> #9 0xffffffff80fdf3b1 in VOP_READ_APV (vop=<value optimized out>,
> a=<value optimized out>) at vnode_if.c:930
> #10 0xffffffff80b461b7 in vn_read (fp=0xfffff8002e1389b0,
> uio=0xfffffe100c459ab0, active_cred=<value optimized out>,
> flags=<value optimized out>, td=0x0) at vnode_if.h:384
> #11 0xffffffff80b425ea in vn_io_fault (fp=0xfffff8002e1389b0,
> uio=0xfffffe100c459ab0, active_cred=0x0, flags=0, td=0x0)
> at /usr/src/sys/kern/vfs_vnops.c:1167
> #12 0xffffffff80ae1525 in dofileread (td=0xfffff804bb5f8940, fd=3,
> fp=0xfffff8002e1389b0, auio=0xfffffe100c459ab0,
> offset=<value optimized out>, flags=Unhandled dwarf expression
> opcode 0x93
> ) at file.h:296
> #13 0xffffffff80ae1228 in kern_readv (td=0xfffff804bb5f8940,
> fd=Unhandled dwarf expression opcode 0x93
> )
> at /usr/src/sys/kern/sys_generic.c:272
> #14 0xffffffff80ae11b3 in sys_read (td=0x0, uap=<value optimized out>)
> at /usr/src/sys/kern/sys_generic.c:185
> #15 0xffffffff80e968da in amd64_syscall (td=0xfffff804bb5f8940,
> traced=0)
> at subr_syscall.c:133
> #16 0xffffffff80e767bb in Xfast_syscall ()
> at /usr/src/sys/amd64/amd64/exception.S:395
> #17 0x00000008009638fa in ?? ()
> Previous frame inner to this frame (corrupt stack?)
> Current language: auto; currently minimal
> (kgdb)
> I have BOTH cores.
--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 214-642-9640 E-Mail: ler at lerctr.org
US Mail: 108 Turvey Cove, Hutto, TX 78634-5688
More information about the freebsd-current
mailing list