hast, zfs, unable to flush disk cache

[Mikolaj Golub]

On Fri, Mar 01, 2013 at 11:39:23AM -0600, Chad M Stewart wrote:

> I've setup a 2 node cluster using HAST.  I'd previously had this
> running 9.0 and then rebuilt the nodes using 9.1.  Under 9.0 I used
> vfs.zfs.vdev.bio_flush_disable=1 but that setting does not appear to
> work in 9.1.  The other difference, previous build had root disk on
> UFS, while this build has only ZFS based file systems.
> 
> FreeBSD node1.san 9.1-RELEASE FreeBSD 9.1-RELEASE #0 r243826: Tue Dec  4 06:55:39 UTC 2012     root at obrian.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  i386
> 
> 
> Mar  1 17:07:25 node1 hastd[1446]: [disk5] (primary) Unable to flush disk cache on activemap update: Operation not supported by device.
> Mar  1 17:07:28 node1 hastd[1440]: [disk3] (primary) Unable to flush disk cache on activemap update: Operation not supported by device.
> Mar  1 17:07:28 node1 hastd[1437]: [disk2] (primary) Unable to flush disk cache on activemap update: Operation not supported by device.
> Mar  1 17:07:28 node1 hastd[1434]: [disk1] (primary) Unable to flush disk cache on activemap update: Operation not supported by device.
> Mar  1 17:07:28 node1 hastd[1446]: [disk5] (primary) Unable to flush disk cache on activemap update: Operation not supported by device.
> Mar  1 17:07:28 node1 hastd[1443]: [disk4] (primary) Unable to flush disk cache on activemap update: Operation not supported by device.
> 
> I tried setting zfs set zfs:zfs_nocacheflush=1 <pool> but that
> diddn't appear to work either.  I get a lot of lines in my log file
> because of this.  I have also tried zfs set sync=disabled <pool> but
> HAST still outputs those lines in the log file.

These flushes were generated by HAST itself when it tried to flush
activemap updates, that is why disabling BIO_FLUSH for ZFS dis not
help much.

Setting "metaflush off" in hast.conf should help though.

BTW, hastd tries to detect devices that do not support BIO_FLUSH,
checking for the returned errno, and automatically disable flushes if
the errno is EOPNOTSUPP (Operation not supported). Unfortunately, your
device returned ENODEV (Operation not supported by device).

What device do you have?

Pawel, do you think it would be a good idea to automatically disable
activemap flush for ENODEV case too?

-- 
Mikolaj Golub