ZFS Panic after freebsd-update

[Jeremy Chadwick]

On Mon, Jul 01, 2013 at 02:04:24PM -0400, Scott Sipe wrote:
> On Mon, Jul 1, 2013 at 1:04 PM, Jeremy Chadwick <jdc at koitsu.org> wrote:
> 
> > On Mon, Jul 01, 2013 at 12:23:45PM -0400, Paul Mather wrote:
> > > On Jul 1, 2013, at 11:49 AM, Jeremy Chadwick <jdc at koitsu.org> wrote:
> > >
> > > > Of course when I see lines like this:
> > > >
> > > >  Trying to mount root from zfs:zroot
> > > >
> > > >  ...this greatly diminishes any chances of "live debugging" on the
> > > >  system.  It amazes me how often I see this come up on the lists --
> > people
> > > >  who have ZFS problems but use ZFS for their root/var/tmp/usr.  I wish
> > > >  that behaviour would stop, as it makes debugging ZFS a serious PITA.
> > > >  This comes up on the list almost constantly, sad panda.
> > >
> > >
> > > I'm not sure why it amazes you that people are making widespread use of
> > ZFS.
> >
> > It's not widespread use of ZFS.  It's widespread use of ZFS as their
> > sole filesystem (specifically root/var/tmp/usr, or more specifically
> > just root/usr).  People are operating with the belief that "ZFS just
> > works", when reality shows "it works until it doesn't".  The mentality
> > seems to be "it's so rock solid it'll never break" along with "it can't
> > happen to me".  I tend to err on the side of caution, hence avoidance of
> > ZFS for critical things like the aforementioned.
> >
> > It's different if you have a UFS root/var/tmp/usr and ZFS for everything
> > else.  You then have a system you can boot/use without issue even if ZFS
> > is crapping the bed.
> >
> 
> 
> > ...
> >
> 
> 
> > 95% of FreeBSD users cannot debug kernel problems**.  To debug a kernel
> > problem, you need: a crash dump, a usable system with the exact
> > kernel/world where the crash happened (i.e. you cannot crash 8.4 ZFS and
> > boot into 8.2 and reliably debug it using that), and (most important of
> > all) a developer who is familiar with kernel debugging *and* familiar
> > with the bits which are crashing.  Those who say what you're quoting are
> > often the latter.
> >
> 
> 
> > ...
> >
> 
> 
> > But the OP is running -RELEASE, and chooses to run that, along with use
> > of freebsd-update for binary updates.  Their choices are limited: stick
> > with 8.2, switch to stable/X, cease use of ZFS, or change OSes entirely.
> >
> 
> So I realize that neither 8.2-RELEASE or 8.4-RELEASE are stable, but I
> ultimately wasn't sure where the right place to go for discuss 8.4 is?

For filesystem issues, freebsd-fs@ is usually the best choice, because
it discusses filesystem-related thing (regardless of stable vs. release,
but knowing what version you have of course is mandatory).

freebsd-stable@ is mainly for stable/X related discussions.

Sorry to add pedanticism to an already difficult situation for you (and
I sympathise, particularly since the purpose of the lists is often
difficult to discern, even with their terse descriptions in mailman).

> Beyond the FS mailing list, was there a better place for my question? I'll
> provide the other requested information (zfs outputs, etc) to wherever
> would be best.

Nope, not as far as I know.  The only other place is send-pr(1), once
you have an issue that can be reproduced.

Keep in mind, however, that none of these options (mailing lists,
send-pr, etc.) mandate a response from anyone.  You/your business (see
below) should be aware that there is always the possibility no one can
help solve the actual problem; as such it's important that companies
have proper upgrade/migration paths, rollback plans, and so on.

> This is a production machine (has been since late 2010) and after tweaking
> some ZFS settings initially has been totally stable. I wasn't incredibly
> closely involved in the initial configuration, but I've done at least one
> binary freebsd-update previously.

Well regardless it sounds like moving from 8.2-RELEASE to 8.4-RELEASE
causes ZFS to break for you, so that would classify as a regression.
What the root cause is, however, is still unknown.

Point: 8.2-RELEASE came out in February 2011, and 8.4-RELEASE came out
in June 2013 -- that's almost 2.5 years of changes between versions.
The number of changes between these two is major -- hundreds, maybe
thousands.  ZFS got worked on heavily during this time as well.

I tend to tell anyone using ZFS that they should be running a stable/X
(particularly stable/9) branch.  I can expand on that justification if
needed, as it's well-founded for a lot of reasons.

> Before this computer I had always done source upgrades. ZFS (and the
> thought of a panic like the one I saw this weekend!) made me leery of doing
> that. We're a small business--we have this server, an offsite backup
> server, and a firewall box. I understand that issues like this are are
> going to happen when I don't have a dedicated testing box, I just like to
> try to minimize them and keep them to weekends!

Understood.

> It sounds like my best bet might be to add a new UFS disk, do a clean
> install of 9.1 onto that disk, and then import my existing ZFS pool?

I would suggest starting with this:

Get stable/9 from the place I mentioned, burn an ISO or dd a memstick
image to a USB flash drive, and then drop to the Fixit shell (or
whatever it's called now, I forget), then try to import your pool.

If it works, then you know migrating to stable/9 should work for you
(i.e. that the bug/issue/whatever is likely fixed in stable/9), or that
some settings/stuff you've configured (probably /boot/loader.conf -- yes
I am quite aware of the screwing about required there for ZFS on 8.2)
on 8.2 is no longer needed or causing problems on 8.4.

If it doesn't work, then you/your company need to work out what your
next choice of action should be.

-- 
| Jeremy Chadwick                                   jdc at koitsu.org |
| UNIX Systems Administrator                http://jdc.koitsu.org/ |
| Making life hard for others since 1977.             PGP 4BD6C0CB |