Dreadful gmirror performance, though each half works fine

[Zaphod Beeblebrox]

On Thu, Apr 17, 2008 at 4:01 PM, Pete French <petefrench at ticketswitch.com>
wrote:

> > In the end we found that ggate was crashy after a week or two of heavy
> use,
> > too... dispite it's performance problems (which can be somewhat fixed by
> > telling gmirror to only read from the local disk)
>
> That last part interests me - how did you manage to make it do that ?
> I read the man page, and the 'prefer' balancing algorithm should
> let you tell it which disc to read from - but there is no mway to
> change the priority on a disc in amirror that I can see. It can only
> be set when inserting new drives. The ddefault is '0' and hence it's
> nbot possible to attach a new drive with a priority below that of
> the existing local drive. I tried using '-1' as a priority to fix
> this, but it came up as 255.
>

I would suppose you might have to sync the mirror and then break off and
forget the local copy and then sync again.  In our case, I'm not sure --- it
was awhile ago, but a number of them are also in the 'load' state --- as the
higher latency network drives would normally show a higher load.


> > Certainly ZFS needs lots of memory --- most of my systems running ZFS
> have
> > 4G of RAM and are running in 64 bit mode.  With the wiki's recomendation
> of
> > a large number of kernel pages, I havn't had a problem with crashing.  I
> am
> > using ZFS RAIDZ as a large data store and ZFS mirroring (separately) on
> my
> > workstation as /usr, /var, and home directories.
>
> All out machine ateb64 bit with between 4 and 16 gig of RAm too, so I
> could
> try that. So you trust it then ? I;d be interested to know exactly which
> options from the wiki page you ended up using for both kernel pages and
> ZFS
> itself. That would be my ideal solution if it is stable enough.
>

Hmm.  Trust is a funny thing.  First the options.  On my notebook:

vm.kmem_size_max="1073741824"
vm.kmem_size="1073741824"

On the 32 bit version, I have options KVA_PAGES=512, but the same
loader.conf settings above.

My large fileserver is used as SMB and NFS filestore for large datasets
generally ending in .avi or .iso.  Not terribly stressed, I don't think.  My
laptop is used as a workstation.

I don't think I'd run mysql or postgresql on zfs yet --- or if I did, I
might run solaris.  It's newer there.  It would make me nervous at any
rate.  My laptop is a core-2-duo Extreme 7900 with 4 gig of ram.  It will
run "make -j8 world" quite quickly on zfs --- and even quicker the 2nd
time.  I've been running /usr, /var and home directories on zfs for several
months now and I havn't had any problems.

That said, I take regular (daily) snapshots and I "zfs send" the snapshots
to the other zfs array for backup.  This is a big advantage to zfs ---
snapshots and snapshot backups are fast (and still checksum protected).

Now as to trust:  ZFS is copy-on-write.  From my reading of problems people
have, it seems that new data might be corrupted in some manner (there are
posts even today about zfs and bittorrent) but the snapshots should not be
affected.  I hedge my bets further by keeping backups.


> > efficient.  Removing the read load from the ggated drive seems to help
> quite
> > a bit in overall performance.  But even with this change, I still found
> that
> > ggate would crash after several days to a week of heavy use.
>
> Well, I upped the networking buffers and queue sizes to what I woulkd
> normally consider 'stupid' values, and now it seems to have settled down
> and is performing well (am using then 'load' balancing algorithm). Shall
> see if it stays that way for the next few weeks given what you have just
> said. I should probably try ZFS on it too, just for my own curiosity.


'load' should almost always prefer the local drive.  Large buffers are
required to compensate for network latency.  Sounds about normal.