zfs upgrade hang

[Teske, Devin]

On Sep 20, 2013, at 5:27 PM, Teske, Devin wrote:

> Hi,
> 
> Seem to be having an issue with "zfs upgrade" hanging.
> 
> Please note that "zpool upgrade" seems to be fine... it's "zfs upgrade" that hangs.
> 
> The system is 8.4-STABLE @ r255470M amd64.
> 
> The dataset versions prior to upgrade are 3 and after upgrade are 5.
> 
> It doesn't always hang. But once it does, it's always in the following ^T state:
> 
> 	tx->tx_sync_done_cv
> 
> You can let it sit there for minutes or hours, but it never completes or enters a
> different state. Also, you can't Ctrl-C it, you can't Ctrl-Z it, you can't kill it, not even
> with `-9'. Further, anything like "zfs list" will hang. All the meanwhile, the filesystem
> is readable and fine.
> 
> The sure-fire way to hit this for us is to attempt a "-a" or "-r" or "-ra" to do many datasets
> at once.
> 
> However, doing one dataset at a time will work... until it too leads to the same state.
> 
> Once we hit this state (hung upgrade) we have to reboot.
> 
> We've been able to get through all the datasets on a box by doing one-at-a-time and
> rebooting when one hangs and ends up in this state but it's frustrating because we can
> usually only do a handful at a time before hitting the problem.
> 
> Scripting it won't help.
> 
> Also, we've tried unmounting the filesystems prior to upgrade too, that didn't help.
> Updating libraries/binaries to r255747 didn't seem to help either. I guess next step is to
> update the kernel to latest stable/8 (which is probably not far ahead of r255470).
> 
> Advice?

Before you chime-in, I think I might have more to add to the puzzle.

It would seem that the LSI-RAID1 pool is the problem.

We always hang when trying to execute "zfs upgrade LSI-RAID1" despite the
fact that we've done a "zfs upgrade" of everything underneath it.

We've also done a "zfs upgrade" of other pools and their descendants on the
same system without this hang.

So I was thinking... what makes the "LSI-RAID1" pool different from, say, the
"NEC1_POOL_A" pool.

The answer is mount-point.

LSI-RAID1 does not have a mountpoint, while everything else does.

Here's the layout we have:

hvm2b# zfs get version
NAME                                            PROPERTY  VALUE    SOURCE
LSI-RAID1                                       version   3        -
LSI-RAID1/vm                                    version   5        -
LSI-RAID1/vm/golden0                            version   5        -
LSI-RAID1/vm/golden0 at pre-cfg0-snap1             version   3        -
LSI-RAID1/vm/golden0 at non-cfg0-snap1             version   3        -
LSI-RAID1/vm/golden0 at zxfer_4709_20130905025825  version   3        -
LSI-RAID1/vm/ipu0c                              version   5        -
LSI-RAID1/vm/ipu1c                              version   5        -
LSI-RAID1/vm/ipu2c                              version   5        -
LSI-RAID1/vm/oos0c                              version   5        -
LSI-RAID1/vmbak                                 version   5        -
LSI-RAID1/vmbak/vm                              version   5        -
NEC2_POOL_B                                     version   5        -
NEC2_POOL_B/oos0c                               version   5        -


As you can see, while trying to work around this hang, we've been able to
upgrade all the datasets with the exception of the one culprit (at the top).

So...

Did I discover a bug? Perhaps relating to "zfs upgrade" touching datasets that
don't have a mountpoint set?
-- 
Devin

_____________
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.