Broken ZFS filesystem

[Freddie Cash]

On Tue, May 8, 2012 at 10:23 AM, Freddie Cash <fjwcash at gmail.com> wrote:
> I have an interesting issue with one single ZFS filesystem in a pool.
> All the other filesystems are fine, and can be mounted, snapshoted,
> destroyed, etc.  But this one filesystem, if I try to do any operation
> on it (zfs mount, zfs snapshot, zfs destroy, zfs set <anything>), it
> spins the system until all RAM is used up (wired), and then hangs the
> box.  The zfs process sits in tx -> tx_sync_done_cv state until the
> box locks up.  CTRL+T of the process only ever shows this:
>    load: 0.46  cmd: zfs 3115 [tx->tx_sync_done_cv)] 36.63r 0.00u 0.00s 0% 2440k
>
> Anyone come across anything similar?  And found a way to fix it, or to
> destroy the filesystem?  Any suggestions on how to go about debugging
> this?  Any magical zdb commands to use?
>
> The filesystem only has 5 MB of data in it (log files), compressed via
> LZJB for a compressratio of ~6x.  There are no snapshots for this
> filesystem.
>
> Dedupe is enabled on the pool and all filesystems.
After more fiddling, testing, and experimenting, it all came down to
not enough RAM in the box to mount the 5 MB filesystem.  After
installing an extra 8 GB of RAM (32 GB total), everything mounted
correctly.  Took 27 GB of wired kernel memory (guessing ARC space) to
do it.

Unmount, mount, export, import, change properties all completed
successfully.  And the box is running correctly with 24 GB of RAM
again.

We'll be ordering more RAM for our ZFS boxes, now.  :)

-- 
Freddie Cash
fjwcash at gmail.com