HAST + ZFS: no action on drive failure

[Mikolaj Golub]

On Thu, 30 Jun 2011 20:02:19 -0700 Timothy Smith wrote:

 TS> First posting here, hopefully I'm doing it right =)

 TS> I also posted this to the FreeBSD forum, but I know some hast folks monitor
 TS> this list regularly and not so much there, so...

 TS> Basically, I'm testing failure scenarios with HAST/ZFS. I got two nodes,
 TS> scripted up a bunch of checks and failover actions between the nodes.
 TS> Looking good so far, though more complex that I expected. It would be cool
 TS> to post it somewher to get some pointers/critiques, but that's another
 TS> thing.

 TS> Anyway, now I'm just seeing what happens when a drive fails on primary node.
 TS> Oddly/sadly, NOTHING!

 TS> Hast just keeps on a ticking, and doesn't change the state of the failed
 TS> drive, so the zpool has no clue the drive is offline. The
 TS> /dev/hast/<resource> remains. The hastd does log some errors to the system
 TS> log like this, but nothing more.

 TS> messages.0:Jun 30 18:39:59 nas1 hastd[11066]: [ada6] (primary) Unable to
 TS> flush activemap to disk: Device not configured.
 TS> messages.0:Jun 30 18:39:59 nas1 hastd[11066]: [ada6] (primary) Local request
 TS> failed (Device not configured): WRITE(4736512, 512).

Although the request to local drive failed it succeeded on remote node, so
data was not lost, it was considered as successful, and no error was returned
to ZFS.

 TS> So, I guess the question is, "Do I have to script a cronjob to check for
 TS> these kinds of errors and then change the hast resource to 'init' or
 TS> something to handle this?" Or is there some kind of hastd config setting
 TS> that I need to set? What's the SOP for this?

Currently the only way to know is monitoring logs. It is not difficult to hook
event for these errors in the HAST code (like it is done for
connect/disconnect, syncstart/done etc) so one could script what to do on an
error occurrence but I am not sure it is a good idea -- the errors may be
generated with high rate.

 TS> As something related too, when the zpool in FreeBSD does finally notice that
 TS> the drive is missing because I have manually changed the hast resource to
 TS> INIT (so the /dev/hast/<res> is gone), my zpool (raidz2) hot spare doesn't
 TS> engage, even with "autoreplace=on". The zpool status of the degraded pool
 TS> seems to indicate that I should manually replace the failed drive. If that's
 TS> the case, it's not really a "hot spare". Does this mean the "FMA Agent"
 TS> referred to in the ZFS manual is not implemented in FreeBSD?

 TS> thanks!
 TS> _______________________________________________
 TS> freebsd-stable at freebsd.org mailing list
 TS> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
 TS> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"

-- 
Mikolaj Golub