Dreadful gmirror performance, though each half works fine

[Zaphod Beeblebrox]

On Thu, Apr 17, 2008 at 2:25 PM, Pete French <petefrench at ticketswitch.com>
wrote:

> > I have experimented with this rather extensively and have operationally
> > decided not to use ggated in combination with gmirror --- it doesn't
> appear
> > to work as well as one might expect.
>
> Ah, thats unmfortunate :-( I oroginally started off using the
> iscsi initiator and target, which did work O.K., but when actually
> ran live ended up locking up after several hours,a nd then panicing
> the kernel. So not ideal - but when it was working it was fine. ggated
> seems the opposite - doesnt crash, but performance is not suitable for any
> kind of real use.
>

In the end we found that ggate was crashy after a week or two of heavy use,
too... dispite it's performance problems (which can be somewhat fixed by
telling gmirror to only read from the local disk)

> I'm somewhat vaguely wondering if zfs with one local and one ggated disk
> > will work well.
>
> I tried ZFS for a while myself, and it works O.K., but has a tendency
> to panic if it wants memory which it can't get. Despite the many different
> guides available, I never managed to get it to the point where I would
> be happy to use it on a production system without worrying about it
> suddenly becomming memory hungry and dieing.


Certainly ZFS needs lots of memory --- most of my systems running ZFS have
4G of RAM and are running in 64 bit mode.  With the wiki's recomendation of
a large number of kernel pages, I havn't had a problem with crashing.  I am
using ZFS RAIDZ as a large data store and ZFS mirroring (separately) on my
workstation as /usr, /var, and home directories.


> Thanks for the inout though - I am doing some more experimentation
> with ggate (basically raing some buffers as per a thread I found) and
> seeing if that helps.
>
> BTW, I think ggate is the problem and not gmirror here - gmirror on top
> of iscsi works fine as I said.
>

I would agree... save the fact that it may be an interaction between the two
and/or UFS that is causing the problems.  Certainly gmirror on local disks
works fine (I've run gmirror/gstripe combinations for several years now as
RAID 10 store with UFS on top).  This is all going to be latency sensitive
--- ggate needs to allow a larger number of oustanding transactions to be
efficient.  Removing the read load from the ggated drive seems to help quite
a bit in overall performance.  But even with this change, I still found that
ggate would crash after several days to a week of heavy use.