Importing spool wedges machine hard

[Steven Schlansker]

Hi all,

I have a backup server running

FreeBSD d0028.nessops.net 9.0-RELEASE FreeBSD 9.0-RELEASE #0: Tue Jan  3 07:46:30 UTC 2012     root at farrell.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  amd64

There is a single ZFS pool used for storage, configured as such:

  pool: tank
    id: 13753647290422885969
 state: ONLINE
action: The pool can be imported using its name or numeric identifier.
config:

        tank          ONLINE
          raidz2-0    ONLINE
            aacd1p1   ONLINE
            aacd2p1   ONLINE
            aacd3p1   ONLINE
            aacd4p1   ONLINE
            aacd5p1   ONLINE
            aacd6p1   ONLINE
            aacd7p1   ONLINE
            aacd8p1   ONLINE
          raidz2-1    ONLINE
            aacd9p1   ONLINE
            aacd10p1  ONLINE
            aacd11p1  ONLINE
            aacd12p1  ONLINE
            aacd13p1  ONLINE
            aacd14p1  ONLINE
            aacd15p1  ONLINE
            aacd16p1  ONLINE

The setup was running just fine for about a month and a half until yesterday when the machine hung hard.  No problem, reset, comes back fine.  Few hours later, it crashes again.  Now any attempt to import the pool spins the disks for about 15-20 minutes and then wedges the machine.  Before it hangs, "top" reports that the zpool command is dancing between active (CPU0), tx->tx, and bio states.  Then the machine becomes unresponsive (both over the network and at the console) and must be reset.


I'm seeing some old diagnostics instructions from http://www.mail-archive.com/zfs-discuss@opensolaris.org/msg18818.html including running zdb -e -d and zdb -e -b, and they are running right now.

But I'm hopeful someone has more concrete advice, as this is a pretty important system to me.

Thanks!
Steven