ZFS mount volume freeze

[Alexandre Biancalana]

Hi list,

 I'm using FreeBSD-8.3-RELEASE-p7 amd64, actually this is a FreeNAS
8.3.1-RELEASE but everything described here were also tested booting
machine with a usb memstick using
FreeBSD-10.0-CURRENT-amd64-20130623-r252101-memstick image with the same
results.

 I've a home file server where I store my personal files, backups and
virtual machine images served through nfs to a XCP. Yesterday my VMs
stopped and all commands (ls, find) issued to the VMs volume (through NFS
or local on fileserver) started to freeze hanging forever.
 Then I reboot the machine and it starts to hang after zpool import and
before volumes get mounted. This machine has 8GB RAM and AMD Bulldozer
six-core CPU.

 I have two zpool:

$ zpool status
  pool: datastore0
 state: ONLINE
  scan: resilvered 834G in 41h27m with 0 errors on Fri Jun 14 15:59:54 2013
config:

        NAME                                            STATE     READ
WRITE CKSUM
        datastore0                                      ONLINE       0
0     0
          mirror-0                                      ONLINE       0
0     0
            gptid/d761e927-d2e2-11e2-bddc-902b34f47b09  ONLINE       0
0     0
            gptid/daa3440d-d3c8-11e2-8405-902b34f47b09  ONLINE       0
0     0

errors: No known data errors

  pool: datastore1
 state: ONLINE
  scan: scrub repaired 1.63G in 1h47m with 0 errors on Thu Jun 27 01:06:59
2013
config:

        NAME                                          STATE     READ WRITE
CKSUM
        datastore1                                    ONLINE       0
0     0
          mirror-0                                    ONLINE       0
0     0
            ada2                                      ONLINE       0
0     0
            ada4                                      ONLINE       0
0     0
        logs
          gptid/74f920fd-d301-11e2-8405-902b34f47b09  ONLINE       0
0     0

errors: No known data errors


$ zfs list
NAME                              USED  AVAIL  REFER  MOUNTPOINT
datastore0                       1.21T  33.7G  2.12M  /mnt/datastore0
datastore0/ale                    534G  33.7G  9.10G  /mnt/datastore0/ale
datastore0/ale/backup             525G  33.7G   143G
/mnt/datastore0/ale/backup
datastore0/forensics             42.1G  33.7G  42.1G
/mnt/datastore0/forensics
datastore0/jail_plugins          1.54G   593M  1.42G
/mnt/datastore0/jail_plugins
datastore0/jail_plugins_archive   278M   746M   278M
/mnt/datastore0/jail_plugins_archive
datastore0/mirror                 520G  33.7G   387G  /mnt/datastore0/mirror
datastore1                        291G   171G  2.29M  /datastore1
datastore1/iso                   51.1G  28.9G  51.1G  /tmp/iso
datastore1/vms                    238G   144G  55.6G  /datastore1/vms

  datastore0 => mirror 2 1TB SATA
  datastore1 => mirror 2 500G SATA

So I boot in single mode and can successfully import both pools, datastore0
is working perfectly I can mount all their volumes and access all data
without any problem.

The problem is with a specify volume where I store the VM files is located
in datastore1/vms, when I try to mount this volume, no matter if I try todo
zfs mount -a or zfs mount datastore1/vms the command hangs forever. I've
another volume in that zpool (datastore1/iso) that I can mount and use
without any problem.

I already tried to do a scrub, that corrected some data but even after that
I can mount that volume.

Now I'm with the machine online with all volume mounted but datastore1/vms
and waiting a zdb datastore1 to finish, (I can provide the cmd output too).

How this machine is my home file server I'm free to do almost everything to
troubleshoot that.

Any ideas on how troubleshoot that further ?

Regards,
Alexandre