GEOM_MIRROR crashes

[Joe Schaefer]

I added two 143G drives da1 and da2 in a round-robin
gmirror array, and ever since the box crashes periodically.
It's been happening over the last few weeks, with increasing
frequency (twice today).

==================================================

/var/log/messages contains

  GEOM_MIRROR: Request failed (error=5). da1[WRITE(offset=117082004992,
    length=16384)]
  GEOM_MIRROR: Device gm0: provider da1 disconnected.
  GEOM_MIRROR: Request failed (error=5). da1[WRITE(offset=117851348480,
    length=16384)]
  GEOM_MIRROR: Request failed (error=5). da2[WRITE(offset=117851348480,
    length=16384)]
  GEOM_MIRROR: Device gm0: provider da2 disconnected.
  GEOM_MIRROR: Device gm0: provider mirror/gm0 destroyed.
  GEOM_MIRROR: Request failed (error=6). da1[WRITE(offset=117851381248,
    length=16384)]
  GEOM_MIRROR: Request failed (error=6). da2[WRITE(offset=117851381248,
    length=16384)]
  ...
  g_vfs_done():mirror/gm0s1b[WRITE(offset=79196635136, length=16384)]error = 6
  g_vfs_done():mirror/gm0s1b[WRITE(offset=79197077504, length=16384)]error = 6
  GEOM_MIRROR: Request failed (error=6). da1[WRITE(offset=120549088768,
    length=16384)]
  GEOM_MIRROR: Request failed (error=6). da2[WRITE(offset=120549088768,
    length=16384)]
  GEOM_MIRROR: Device gm0 destroyed.
  g_vfs_done():mirror/gm0s1b[WRITE(offset=81894342656, length=16384)]error = 6
  g_vfs_done():mirror
  /gm0s1b[READ(offset=53400334336, length=2048)]error = 6 
  <crash>

The subsequent backtrace with kgdb is

#0  doadump () at pcpu.h:165
#1  0xc063aaeb in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:399
#2  0xc063ae11 in panic (fmt=0xc0875357 "vinvalbuf: dirty bufs")
    at /usr/src/sys/kern/kern_shutdown.c:555
#3  0xc0690739 in bufobj_invalbuf (bo=0xc393ab60, flags=1, td=0xc380c480, 
    slpflag=0, slptimeo=0) at /usr/src/sys/kern/vfs_subr.c:1000
#4  0xc0690979 in vinvalbuf (vp=0xc393aaa0, flags=1, td=0xc380c480, slpflag=0, 
    slptimeo=0) at /usr/src/sys/kern/vfs_subr.c:1067
#5  0xc0693081 in vgonel (vp=0xc393aaa0) at /usr/src/sys/kern/vfs_subr.c:2399
#6  0xc0692fb2 in vgone (vp=0xc393aaa0) at /usr/src/sys/kern/vfs_subr.c:2356
#7  0xc05ed9c4 in devfs_delete (dm=0xc3839300, de=0xc383f300)
    at /usr/src/sys/fs/devfs/devfs_devs.c:243
#8  0xc05edbc0 in devfs_populate_loop (dm=0xc3839300, cleanup=0)
    at /usr/src/sys/fs/devfs/devfs_devs.c:350
#9  0xc05eddfa in devfs_populate (dm=0xc3839300)
    at /usr/src/sys/fs/devfs/devfs_devs.c:446
#10 0xc05ef9ff in devfs_lookupx (ap=0x0)
    at /usr/src/sys/fs/devfs/devfs_vnops.c:512
#11 0xc05efb3b in devfs_lookup (ap=0xe6bcd9c8)
    at /usr/src/sys/fs/devfs/devfs_vnops.c:576
#12 0xc081f78f in VOP_LOOKUP_APV (vop=0xc08c4500, a=0xe6bcd9c8)
    at vnode_if.c:99
#13 0xc068b969 in lookup (ndp=0xe6bcdbcc) at vnode_if.h:56
#14 0xc068b2aa in namei (ndp=0xe6bcdbcc) at /usr/src/sys/kern/vfs_lookup.c:203
#15 0xc069d8e9 in vn_open_cred (ndp=0xe6bcdbcc, flagp=0xe6bcdccc, cmode=0, 
    cred=0xc34bda80, fdidx=6) at /usr/src/sys/kern/vfs_vnops.c:182
#16 0xc069d63a in vn_open (ndp=0x0, flagp=0xe6bcdccc, cmode=0, fdidx=6)
    at /usr/src/sys/kern/vfs_vnops.c:91
#17 0xc0696642 in kern_open (td=0xc380c480, path=0x0, pathseg=UIO_USERSPACE, 
    flags=6, mode=0) at /usr/src/sys/kern/vfs_syscalls.c:979
#18 0xc0696556 in open (td=0xc380c480, uap=0xe6bcdd04)
    at /usr/src/sys/kern/vfs_syscalls.c:945
#19 0xc08101ff in syscall (frame=
      {tf_fs = 59, tf_es = -1078001605, tf_ds = -1078001605, tf_edi = 7,
       tf_esi = -1077947008, tf_ebp = -1077948728, tf_isp = -423830172,
       tf_ebx = 0, tf_edx = 8, tf_ecx = 47, tf_eax = 5, tf_trapno = 0,
       tf_err = 2, tf_eip = 671913971, tf_cs = 51, tf_eflags = 582,
       tf_esp = -1077948852, tf_ss = 59})
    at /usr/src/sys/i386/i386/trap.c:976
#20 0xc07fcd0f in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:200
#21 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)

==================================================

Can anyone tell me what kind of problem I'm running into here?
Did I somehow botch the gmirror setup?  Is this a hardware problem?

TIA.

-- 
Joe Schaefer