Best practice for high availability ZFS pool

Fri Nov 11 16:14:05 UTC 2016

On Fri, Nov 11, 2016 at 04:16:52PM +0100, Palle Girgensohn wrote:
> Hi,
> 
> Pinging this old thread.
> 
> We have revisited this question:
> 
> A simple stable solution for a redundant storage with little or no down time when a machine breaks. Storage is served using NFS only.
> 
> 
> It seems true HA is always complicated. I'd rather go for a simple understandable solution and accept sub minute downtime rather than a complicated solution. For our needs, the pretty solution lined up in the FreeBSD Magazine seems a bit overly complicated.
> 
> So here's what we are pondering:
> 
> - one SAS dual port disk box
> 
> - connect a master host machine to one port and a slave host machine to the the other port
> 
> - one host is MASTER, it serves all requests
> 
> - one host is SLAVE, doing nothing but waiting for the MASTER to fail
> 
> - fail over would be handled with zpool export / zpool import, or just zpool import -F if the master dies.
> 
> - MASTER/SLAVE election and avoiding split brain using for example CARP.
> 
> This is not a real HA solution since zpool import takes about a minute. Is this true for a large array?
> 
> Would this suggestion work?

I'm using someting like this here, a zpool over 2 local disks and 2
iscsi disks and the following failover script:
https://gist.github.com/silenius/cb10171498071bdbf6040e30a0cab5c2

It works like a charm except that I'm having this issue:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=211990 

(apparently this problem does not appear on 11.0-RELEASE)

> 
> Are there better ideas out there?
> 
> Cheers,
> Palle
> 
> 
> 
> 
> 
> 
> > 18 maj 2016 kl. 09:58 skrev Sean Chittenden <sean at chittenden.org>:
> > 
> > https://www.freebsdfoundation.org/wp-content/uploads/2015/12/vol2_no4_groupon.pdf
> > 
> > mps(4) was good to us.  What’s your workload?  -sc
> > 
> > --
> > Sean Chittenden
> > sean at chittenden.org
> > 
> > 
> >> On May 18, 2016, at 03:53 , Palle Girgensohn <girgen at pingpong.net> wrote:
> >> 
> >> 
> >> 
> >>> 17 maj 2016 kl. 18:13 skrev Joe Love <joe at getsomewhere.net>:
> >>> 
> >>> 
> >>>> On May 16, 2016, at 5:08 AM, Palle Girgensohn <girgen at FreeBSD.org> wrote:
> >>>> 
> >>>> Hi,
> >>>> 
> >>>> We need to set up a ZFS pool with redundance. The main goal is high availability - uptime.
> >>>> 
> >>>> I can see a few of paths to follow.
> >>>> 
> >>>> 1. HAST + ZFS
> >>>> 
> >>>> 2. Some sort of shared storage, two machines sharing a JBOD box.
> >>>> 
> >>>> 3. ZFS replication (zfs snapshot + zfs send | ssh | zfs receive)
> >>>> 
> >>>> 4. using something else than ZFS, even a different OS if required.
> >>>> 
> >>>> My main concern with HAST+ZFS is performance. Google offer some insights here, I find mainly unsolved problems. Please share any success stories or other experiences.
> >>>> 
> >>>> Shared storage still has a single point of failure, the JBOD box. Apart from that, is there even any support for the kind of storage PCI cards that support dual head for a storage box? I cannot find any.
> >>>> 
> >>>> We are running with ZFS replication today, but it is just too slow for the amount of data.
> >>>> 
> >>>> We prefer to keep ZFS as we already have a rather big (~30 TB) pool and also tools, scripts, backup all is using ZFS, but if there is no solution using ZFS, we're open to alternatives. Nexenta springs to mind, but I believe it is using shared storage for redundance, so it does have single points of failure?
> >>>> 
> >>>> Any other suggestions? Please share your experience. :)
> >>>> 
> >>>> Palle
> >>> 
> >>> I don’t know if this falls into the realm of what you want, but BSDMag just released an issue with an article entitled “Adding ZFS to the FreeBSD dual-controller storage concept.”
> >>> https://bsdmag.org/download/reusing_openbsd/
> >>> 
> >>> My understanding in this setup is that the only single point of failure for this model is the backplanes that the drives would connect to.  Depending on your controller cards, this could be alleviated by simply using multiple drive shelves, and only using one drive/shelf as part of a vdev (then stripe or whatnot over your vdevs).
> >>> 
> >>> It might not be what you’re after, as it’s basically two systems with their own controllers, with a shared set of drives.  Some expansion from the virtual world to real physical systems will probably need additional variations.
> >>> I think the TrueNAS system (with HA) is setup similar to this, only without the split between the drives being primarily handled by separate controllers, but someone with more in-depth knowledge would need to confirm/deny this.
> >>> 
> >>> -Jo
> >> 
> >> Hi,
> >> 
> >> Do you know any specific controllers that work with dual head?
> >> 
> >> Thanks.,
> >> Palle
> >> 
> >> 
> >> _______________________________________________
> >> freebsd-fs at freebsd.org mailing list
> >> https://lists.freebsd.org/mailman/listinfo/freebsd-fs
> >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"
> > 
> 
> _______________________________________________
> freebsd-fs at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"

-- 
Julien Cigar
Belgian Biodiversity Platform (http://www.biodiversity.be)
PGP fingerprint: EEF9 F697 4B68 D275 7B11  6A25 B2BB 3710 A204 23C0
No trees were killed in the creation of this message.
However, many electrons were terribly inconvenienced.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: not available
URL: <http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20161111/c92bfd45/attachment.sig>