ZFSv28+NFSv4 poor file creation performance, "sync=disabled" has no effect

[Jeremy Chadwick]

On Sat, Sep 03, 2011 at 05:00:05PM -0700, David Brodbeck wrote:
> On Fri, Sep 2, 2011 at 6:36 PM, Rick Macklem <rmacklem at uoguelph.ca> wrote:
> 
> > One post explained how disabling the ZIL can result in up to 5seconds worth
> > of changes being lost if/when the server crashes. (A lot can change on a
> > file
> > system in 5sec. The NFS protocol assumes all fs changes related to a file
> > creation are done before the server replies to the RPC. As such, disabling
> > the
> > ZIL does violate the protocol specs and means you are living dangerously.)
> >
> 
> ...
>
> Also, most other filesystems make no guarantees that data has been
> flushed to the platters, only that it's been sent to the disk
> hardware, so it really isn't a big increase in risk compared to our
> pre-ZFS setup.

Can you explain where you got this impression from?  Please provide
actual references/validation/proof for non-network filesystems.

Disk-level filesystems are more or less at the mercy of the applications
using them; if a programmer calls open(2) without O_FSYNC or O_SYNC, or
does not explicitly flush I/O to disk in their software, then is the
filesystem really to blame?

Furthermore, the world does know of some disks/devices/products where
executing BIO_FLUSH (which on ATA disks should get translated into the
ATA FLUSH or FLUSH EXT command) does not truly write the data to the
platters.  Again: the filesystem is innocent, blame the
disk/device/manufacturer.

> The sync=disabled parameter was, in my understanding, added as a
> per-filesystem way to achieve this risk/performance tradeoff, instead of
> zil_disable, which was system-wide.  I posted the question because I'm not
> seeing any changes when I use sync=disabled.

pjd@, mm@, and others will need to comment on this.  -fs is the best
list for this, so I'm a little surprised no key members have chimed in
here.

What we (the community) need clarification regarding is whether or not
OpenSolaris and/or Illumos still provides the zil_disable tunable on
those OSes; if they do, FreeBSD should provide the same (which means
removable of zil_disable on FreeBSD is effectively a regression).  If
said OSes do not provide it, then FreeBSD should not provide it.  If
said OSes moved to using the sync parameter where it works, yet it does
not work on FreeBSD, then that's a bug and a PR should be filed +
discussion induced.

And yes, I'm aware that I have given the "try using the sync parameter
instead" advice to others here on FreeBSD lists.

> This is a bit of a show-stopper for us because performance with full syncing
> is not acceptable. I agree that it's not ideal, but we don't have the extra
> drive bays to add SSDs at this point; also it seems like people have had a
> lot of problems with SSDs and FreeBSD, so I'd prefer for the technology to
> mature a bit before I risk my entire pool by putting a ZIL on one.

The risks/complexities and performance concerns with SSDs when used as
part of a ZFS filesystem on FreeBSD greatly differ from that of using an
SSD as a dedicated intent log (ZIL) device.

What exactly do you need ZFS for given your requirements?  Is it that
you want a simple-to-use LVM-like filesystem that offers a vast
in-memory cache but without the data integrity bits?

-- 
| Jeremy Chadwick                                jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                   Mountain View, CA, US |
| Making life hard for others since 1977.               PGP 4BD6C0CB |