Bad hardware + zfs = panic

[Pawel Jakub Dawidek]

On Wed, May 12, 2010 at 12:21:56PM +0200, Pawel Jakub Dawidek wrote:
> On Thu, May 06, 2010 at 03:22:17AM +0200, Ståle Kristoffersen wrote:
> > I've been debugging a hardware error for the past few days, and I think it
> > was the CPU and that it is now fixed. But reading a file that was written to a
> > zfs-pool when stuff got corrupted still triggered a panic in ZFS code:
> > 
> > Fatal trap 12: page fault while in kernel mode
> > cpuid = 0; apic id = 00
> > fault virtual address   = 0x28
> > fault code              = supervisor read data, page not present
> > instruction pointer     = 0x20:0xffffffff8106f2d3
> > stack pointer           = 0x28:0xffffff80774914e0
> > frame pointer           = 0x28:0xffffff8077491510
> > code segment            = base 0x0, limit 0xfffff, type 0x1b
> >                         = DPL 0, pres 1, long 1, def32 0, gran 1
> > processor eflags        = interrupt enabled, resume, IOPL = 0
> > current process         = 1350 (smbd)
> > trap number             = 12
> > panic: page fault
> > cpuid = 0
> > Uptime: 2m53s
> > 
> > The lines in the backtrace that got my attention was:
> > #6  0xffffffff80847c73 in calltrap () at
> > /usr/src/sys/amd64/amd64/exception.S:224
> > #7  0xffffffff8106f2d3 in vdev_is_dead (vd=0x0) at
> > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev.c:1847
> > #8  0xffffffff8106f2ed in vdev_readable (vd=0x0) at
> > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev.c:1854
> > 
> > The complete bt is available here:
> > http://heim.ifi.uio.no/staalebk/zfs-panic.txt
> > 
> > As you can see vd=0x0, and I think that caused the panic, since it
> > tried to follow that pointer:
> >  return (vd->vdev_state < VDEV_STATE_DEGRADED);
> > 
> > I then tried to remove the file and I got this:
> > Solaris: WARNING: metaslab_free_dva(): bad DVA
> > 199476166:1296607792756162560
> > Solaris: WARNING: metaslab_free_dva(): bad DVA 4236221:7256850009726709760
> > Solaris: WARNING: metaslab_free_dva(): bad DVA
> > 935912721:16480078061480073216
> > 
> > Maybe there should be a test to check if vd was zero, and
> > throw an io-error or something, instead of panicing?
> 
> Well, I don't think it should be possible for vdev to be NULL.
> But if you still have this panic, can you try this patch:
> 
> 	http://people.freebsd.org/~pjd/patches/vdev_mirror.c.patch

It looks like:

	http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6435666

The work-around is to remove /boot/zfs/zpool.cache and import the pool
again.

-- 
Pawel Jakub Dawidek                       http://www.wheelsystems.com
pjd at FreeBSD.org                           http://www.FreeBSD.org
FreeBSD committer                         Am I Evil? Yes, I Am!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 196 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20100512/53e5cddb/attachment.pgp