ZFS Kernel Panic on 10.0-RELEASE
Mike Carlson
mike at bayphoto.com
Fri May 30 19:36:23 UTC 2014
Hey FS@
Over the weekend, we had upgraded one of our servers from 9.1-RELEASE to
10.0-RELEASE, and then the zpool was upgraded (from 28 to 5000)
Tuesday afternoon, the server suddenly rebooted (kernel panic), and as
soon as it tried to remount all of its ZFS volumes, it panic'd again.
We have a zfs on root install, the total raidz is around 9TB.
The volume it is panic-ing on is zroot/data/working, which has (or had
at this point) 4TB of data. I can boot off of the 10.0-RELEASE usb
image, and mount:
zroot
zroot/usr
zroot/var
zroot/data
Just not the large volume that had 4TB of data.
I've set the volume to readonly, and that still causes a panic upon
mount, I was able to snapshot the problematic volume, and even do a send
| receive, but that panics when the transfer is nearly complete (4.64TB
out of 4.639TB)
Now, that data is not super critical, its basically scratch storage
where archives are extracted and shuffled around, and then moved off to
another location. We just like to keep about 60 days worth in case we
need to re-process something.
The more important issue is, why did this happen, and what can we
recover from situations like this. It looks pretty bad when any data is
lost.
See url's for the kernel panic message (used my phone to take a picture)
https://drive.google.com/file/d/0B0i2JyKe_ya2RnRYT3A1Qk5ldkk/edit?usp=sharing
https://drive.google.com/file/d/0B0i2JyKe_ya2YWNlbVl3MVFlTGc/edit?usp=sharing
Here is the zpool history for that storage:
History for 'zroot':
2013-11-19.11:31:37 zpool create -o altroot=/mnt -o
cachefile=/var/tmp/zpool.cache -f zroot raidz2 /dev/gpt/disk0
/dev/gpt/disk1 /dev/gpt/disk2 /dev/gpt/disk3 /dev/gpt/disk4
/dev/gpt/disk5 /dev/gpt/disk6 /dev/gpt/disk7 /dev/gpt/disk8
/dev/gpt/disk9 /dev/gpt/disk11 spare /dev/gpt/disk12
2013-11-19.11:31:37 zpool export zroot
2013-11-19.11:31:38 zpool import -o altroot=/mnt -o
cachefile=/var/tmp/zpool.cache zroot
2013-11-19.11:31:38 zpool set bootfs=zroot zroot
2013-11-19.11:31:43 zfs set checksum=fletcher4 zroot
2013-11-19.11:34:11 zfs create zroot/usr
2013-11-19.11:34:11 zfs create zroot/home
2013-11-19.11:34:11 zfs create zroot/var
2013-11-19.11:34:11 zfs create zroot/data
2013-11-19.11:34:11 zfs create -o compression=on -o exec=on -o
setuid=off zroot/tmp
2013-11-19.11:34:11 zfs create -o compression=lzjb -o setuid=off
zroot/usr/ports
2013-11-19.11:34:11 zfs create -o compression=off -o exec=off -o
setuid=off zroot/usr/ports/distfiles
2013-11-19.11:34:11 zfs create -o compression=off -o exec=off -o
setuid=off zroot/usr/ports/packages
2013-11-19.11:34:11 zfs create -o compression=lzjb -o exec=off -o
setuid=off zroot/usr/src
2013-11-19.11:34:11 zfs create -o compression=lzjb -o exec=off -o
setuid=off zroot/var/crash
2013-11-19.11:34:11 zfs create -o exec=off -o setuid=off zroot/var/db
2013-11-19.11:34:11 zfs create -o compression=lzjb -o exec=on -o
setuid=off zroot/var/db/pkg
2013-11-19.11:34:11 zfs create -o exec=off -o setuid=off zroot/var/empty
2013-11-19.11:34:11 zfs create -o compression=lzjb -o exec=off -o
setuid=off zroot/var/log
2013-11-19.11:34:11 zfs create -o compression=gzip -o exec=off -o
setuid=off zroot/var/mail
2013-11-19.11:34:11 zfs create -o exec=off -o setuid=off zroot/var/run
2013-11-19.11:34:11 zfs create -o compression=lzjb -o exec=on -o
setuid=off zroot/var/tmp
2013-11-19.11:34:11 zfs create -V 4G zroot/swap
2013-11-19.11:34:11 zfs set org.freebsd:swap=on zroot/swap
2013-11-19.11:34:11 zfs set checksum=off zroot/swap
2013-11-19.11:43:24 zfs set readonly=on zroot/var/empty
2013-11-19.11:43:40 zfs set mountpoint=legacy zroot
2013-11-19.11:43:50 zfs set mountpoint=/tmp zroot/tmp
2013-11-19.11:43:58 zfs set mountpoint=/usr zroot/usr
2013-11-19.11:44:05 zfs set mountpoint=/var zroot/var
2013-11-19.11:44:12 zfs set mountpoint=/home zroot/home
2013-11-19.11:44:18 zfs set mountpoint=/data zroot/data
2013-11-19.20:11:53 zfs create zroot/data/working
2013-11-19.20:17:59 zpool scrub zroot
2013-11-19.21:21:23 zfs set aclmode=passthrough zroot/data
2013-11-19.21:21:33 zfs set aclinherit=passthrough zroot/data
2013-11-21.00:58:57 zfs set compression=lzjb zroot/data/working
2014-05-24.14:24:40 zfs set readonly=off zroot/var/empty
2014-05-24.15:37:15 zpool upgrade zroot
2014-05-27.15:32:41 zfs set mountpoint=/mnt zroot
2014-05-27.15:33:55 zfs set mountpoint=/mnt/tmp zroot/tmp
2014-05-27.15:34:03 zfs set mountpoint=/mnt/var zroot/var
2014-05-27.15:34:13 zfs set mountpoint=/mnt/crash zroot/var/crash
2014-05-27.15:34:22 zfs set mountpoint=/mnt/db zroot/var/db
2014-05-27.15:34:35 zfs set mountpoint=/mnt/db/pkg zroot/var/db/pkg
2014-05-27.15:34:47 zfs set mountpoint=/mnt/db/empty zroot/var/empty
2014-05-27.15:35:22 zfs set mountpoint=/mnt/var/db zroot/var/db
2014-05-27.15:35:29 zfs set mountpoint=/mnt/var/db/pkg zroot/var/db/pkg
2014-05-27.15:35:38 zfs set mountpoint=/mnt/var/empty zroot/var/empty
2014-05-27.15:35:45 zfs set mountpoint=/mnt/var/log zroot/var/log
2014-05-27.15:35:54 zfs set mountpoint=/mnt/var/mail zroot/var/mail
2014-05-27.15:36:02 zfs set mountpoint=/mnt/var/run zroot/var/run
2014-05-27.15:36:09 zfs set mountpoint=/mnt/var/tmp zroot/var/tmp
2014-05-27.15:36:34 zfs set mountpoint=/mnt/usr zroot/usr
2014-05-27.15:36:40 zfs set mountpoint=/mnt/usr/ports zroot/usr/ports
2014-05-27.15:36:54 zfs set mountpoint=/mnt/usr/distfiles
zroot/usr/ports/distfiles
2014-05-27.15:37:12 zfs set mountpoint=/mnt/usr/ports/packages
zroot/usr/ports/packages
2014-05-27.15:37:20 zfs set mountpoint=/mnt/usr/ports/distfiles
zroot/usr/ports/distfiles
2014-05-27.15:37:35 zfs set mountpoint=/mnt/usr/src zroot/usr/src
2014-05-27.15:37:53 zfs set mountpoint=/mnt/home zroot/home
2014-05-27.15:38:39 zfs set mountpoint=/mnt/data zroot/data
2014-05-27.15:38:47 zfs set mountpoint=/mnt/data/working
zroot/data/working
2014-05-27.15:57:53 zpool scrub zroot
2014-05-28.09:34:16 zfs snapshot zroot/data/working at 1
2014-05-28.18:55:12 zfs set readonly=on zroot/data/working
The full zfs attributes for that particular volume:
NAME PROPERTY VALUE SOURCE
zroot/data/working type filesystem -
zroot/data/working creation Tue Nov 19 20:11 2013 -
zroot/data/working used 4.64T -
zroot/data/working available 2.49T -
zroot/data/working referenced 4.64T -
zroot/data/working compressratio 1.00x -
zroot/data/working mounted no -
zroot/data/working quota none default
zroot/data/working reservation none default
zroot/data/working recordsize 128K default
zroot/data/working mountpoint /mnt/data/working local
zroot/data/working sharenfs off default
zroot/data/working checksum fletcher4 inherited from zroot
zroot/data/working compression lzjb local
zroot/data/working atime on default
zroot/data/working devices on default
zroot/data/working exec on default
zroot/data/working setuid on default
zroot/data/working readonly on local
zroot/data/working jailed off default
zroot/data/working snapdir hidden default
zroot/data/working aclmode passthrough inherited from
zroot/data
zroot/data/working aclinherit passthrough inherited from
zroot/data
zroot/data/working canmount on default
zroot/data/working xattr on default
zroot/data/working copies 1 default
zroot/data/working version 5 -
zroot/data/working utf8only off -
zroot/data/working normalization none -
zroot/data/working casesensitivity sensitive -
zroot/data/working vscan off default
zroot/data/working nbmand off default
zroot/data/working sharesmb off default
zroot/data/working refquota none default
zroot/data/working refreservation none default
zroot/data/working primarycache all default
zroot/data/working secondarycache all default
zroot/data/working usedbysnapshots 0 -
zroot/data/working usedbydataset 4.64T -
zroot/data/working usedbychildren 0 -
zroot/data/working usedbyrefreservation 0 -
zroot/data/working logbias latency default
zroot/data/working dedup off default
zroot/data/working mlslabel -
zroot/data/working sync standard default
zroot/data/working refcompressratio 1.00x -
zroot/data/working written 0 -
zroot/data/working logicalused 4.65T -
zroot/data/working logicalreferenced 4.65T -
Was this a zfs 10.0-RELEASE issue? Or, did our Dell PERC H710 controller
just happen to become an issue and the timing is coincidental?
Any pointers on either restoring the data or preventing this in the
future would be great
Mike C
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6054 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20140530/2799c49e/attachment.bin>
More information about the freebsd-fs
mailing list