HAST + ZFS + NFS + CARP

krad kraduk at gmail.com
Thu Aug 18 07:36:28 UTC 2016 

I didnt think touch was atomic, mkdir is though

On 18 August 2016 at 08:32, InterNetX - Juergen Gotteswinter <
juergen.gotteswinter at internetx.com> wrote:

>
>
> Am 17.08.2016 um 20:03 schrieb Linda Kateley:
> > I just do consulting so I don't always get to see the end of the
> > project. Although we are starting to do more ongoing support so we can
> > see the progress..
> >
> > I have worked with some of the guys from high-availability.com for maybe
> > 20 years. RSF-1 is the cluster that is bundled with nexenta. Does work
> > beautifully with omni/illumos. The one customer I have running it in
> > prod is an isp in south america running openstack and zfs on freebsd as
> > iscsi. Big boxes, 90+ drives per frame.  If someone would like try it, i
> > have some contacts there. Ping me offlist.
>
> no offense, but it sounds a bit like marketing.
>
> here: running nexenta ha setup since several years with one catastrophic
> failure due to split brain
>
> >
> > You do risk losing data if you batch zfs send. It is very hard to run
> > that real time.
>
> depends on how much data changes aka delta size
>
>
> You have to take the snap then send the snap. Most
> > people run in cron, even if it's not in cron, you would want one to
> > finish before you started the next.
>
> thats the reason why lock files where invented, tools like zrep handle
> that themself via additional zfs properties
>
> or, if one does not trust a single layer
>
> -- snip --
> #!/bin/sh
> if [ ! -f /var/run/replic ] ; then
>         touch /var/run/replic
>         /blah/path/zrep sync all >> /var/log/zfsrepli.log
>         rm -f /var/run/replic
> fi
> -- snip --
>
> something like this, simple
>
>  If you lose the sending host before
> > the receive is complete you won't have a full copy.
>
> if rsf fails, and you end up in split brain you loose way more. been
> there, seen that.
>
> With zfs though you
> > will probably still have the data on the sending host, however long it
> > takes to bring it back up. RSF-1 runs in the zfs stack and send the
> > writes to the second system. It's kind of pricey, but actually much less
> > expensive than commercial alternatives.
> >
> > Anytime you run anything sync it adds latency but makes things safer..
>
> not surprising, it all depends on the usecase
>
> > There is also a cool tool I like, called zerto for vmware that sits in
> > the hypervisor and sends a sync copy of a write locally and then an
> > async remotely. It's pretty cool. Although I haven't run it myself, have
> > a bunch of customers running it. I believe it works with proxmox too.
> >
> > Most people I run into (these days) don't mind losing 5 or even 30
> > minutes of data. Small shops.
>
> you talk about minutes, what delta size are we talking here about? why
> not using zrep in a loop for example
>
>  They usually have a copy somewhere else.
> > Or the cost of 5-30 minutes isn't that great. I used work as a
> > datacenter architect for sun/oracle with only fortune 500. There losing
> > 1 sec could put large companies out of business. I worked with banks and
> > exchanges.
>
> again, usecase. i bet 99% on this list are not operating fortune 500
> bank filers
>
> They couldn't ever lose a single transaction. Most people
> > nowadays do the replication/availability in the application though and
> > don't care about underlying hardware, especially disk.
> >
> >
> > On 8/17/16 11:55 AM, Chris Watson wrote:
> >> Of course, if you are willing to accept some amount of data loss that
> >> opens up a lot more options. :)
> >>
> >> Some may find that acceptable though. Like turning off fsync with
> >> PostgreSQL to get much higher throughput. As little no as you are made
> >> *very* aware of the risks.
> >>
> >> It's good to have input in this thread from one with more experience
> >> with RSF-1 than the rest of us. You confirm what others have that said
> >> about RSF-1, that it's stable and works well. What were you deploying
> >> it on?
> >>
> >> Chris
> >>
> >> Sent from my iPhone 5
> >>
> >> On Aug 17, 2016, at 11:18 AM, Linda Kateley <lkateley at kateley.com
> >> <mailto:lkateley at kateley.com>> wrote:
> >>
> >>> The question I always ask, as an architect, is "can you lose 1 minute
> >>> worth of data?" If you can, then batched replication is perfect. If
> >>> you can't.. then HA. Every place I have positioned it, rsf-1 has
> >>> worked extremely well. If i remember right, it works at the dmu. I
> >>> would suggest try it. They have been trying to have a full freebsd
> >>> solution, I have several customers running it well.
> >>>
> >>> linda
> >>>
> >>>
> >>> On 8/17/16 4:52 AM, Julien Cigar wrote:
> >>>> On Wed, Aug 17, 2016 at 11:05:46AM +0200, InterNetX - Juergen
> >>>> Gotteswinter wrote:
> >>>>>
> >>>>> Am 17.08.2016 um 10:54 schrieb Julien Cigar:
> >>>>>> On Wed, Aug 17, 2016 at 09:25:30AM +0200, InterNetX - Juergen
> >>>>>> Gotteswinter wrote:
> >>>>>>>
> >>>>>>> Am 11.08.2016 um 11:24 schrieb Borja Marcos:
> >>>>>>>>> On 11 Aug 2016, at 11:10, Julien Cigar <julien at perdition.city
> >>>>>>>>> <mailto:julien at perdition.city>> wrote:
> >>>>>>>>>
> >>>>>>>>> As I said in a previous post I tested the zfs send/receive
> >>>>>>>>> approach (with
> >>>>>>>>> zrep) and it works (more or less) perfectly.. so I concur in
> >>>>>>>>> all what you
> >>>>>>>>> said, especially about off-site replicate and synchronous
> >>>>>>>>> replication.
> >>>>>>>>>
> >>>>>>>>> Out of curiosity I'm also testing a ZFS + iSCSI + CARP at the
> >>>>>>>>> moment,
> >>>>>>>>> I'm in the early tests, haven't done any heavy writes yet, but
> >>>>>>>>> ATM it
> >>>>>>>>> works as expected, I havent' managed to corrupt the zpool.
> >>>>>>>> I must be too old school, but I don’t quite like the idea of
> >>>>>>>> using an essentially unreliable transport
> >>>>>>>> (Ethernet) for low-level filesystem operations.
> >>>>>>>>
> >>>>>>>> In case something went wrong, that approach could risk
> >>>>>>>> corrupting a pool. Although, frankly,
> >>>>>>>> ZFS is extremely resilient. One of mine even survived a SAS HBA
> >>>>>>>> problem that caused some
> >>>>>>>> silent corruption.
> >>>>>>> try dual split import :D i mean, zpool -f import on 2 machines
> >>>>>>> hooked up
> >>>>>>> to the same disk chassis.
> >>>>>> Yes this is the first thing on the list to avoid .. :)
> >>>>>>
> >>>>>> I'm still busy to test the whole setup here, including the
> >>>>>> MASTER -> BACKUP failover script (CARP), but I think you can prevent
> >>>>>> that thanks to:
> >>>>>>
> >>>>>> - As long as ctld is running on the BACKUP the disks are locked
> >>>>>> and you can't import the pool (even with -f) for ex (filer2 is the
> >>>>>> BACKUP):
> >>>>>> https://gist.github.com/silenius/f9536e081d473ba4fddd50f59c56b58f
> >>>>>>
> >>>>>> - The shared pool should not be mounted at boot, and you should
> >>>>>> ensure
> >>>>>> that the failover script is not executed during boot time too:
> >>>>>> this is
> >>>>>> to handle the case wherein both machines turn off and/or re-ignite
> at
> >>>>>> the same time. Indeed, the CARP interface can "flip" it's status
> >>>>>> if both
> >>>>>> machines are powered on at the same time, for ex:
> >>>>>> https://gist.github.com/silenius/344c3e998a1889f988fdfc3ceba57aaf
> and
> >>>>>> you will have a split-brain scenario
> >>>>>>
> >>>>>> - Sometimes you'll need to reboot the MASTER for some $reasons
> >>>>>> (freebsd-update, etc) and the MASTER -> BACKUP switch should not
> >>>>>> happen, this can be handled with a trigger file or something like
> >>>>>> that
> >>>>>>
> >>>>>> - I've still have to check if the order is OK, but I think that as
> >>>>>> long
> >>>>>> as you shutdown the replication interface and that you adapt the
> >>>>>> advskew (including the config file) of the CARP interface before the
> >>>>>> zpool import -f in the failover script you can be relatively
> >>>>>> confident
> >>>>>> that nothing will be written on the iSCSI targets
> >>>>>>
> >>>>>> - A zpool scrub should be run at regular intervals
> >>>>>>
> >>>>>> This is my MASTER -> BACKUP CARP script ATM
> >>>>>> https://gist.github.com/silenius/7f6ee8030eb6b923affb655a259bfef7
> >>>>>>
> >>>>>> Julien
> >>>>>>
> >>>>> 100€ question without detailed looking at that script. yes from a
> >>>>> first
> >>>>> view its super simple, but: why are solutions like rsf-1 such more
> >>>>> powerful / featurerich. Theres a reason for, which is that they try
> to
> >>>>> cover every possible situation (which makes more than sense for
> this).
> >>>> I've never used "rsf-1" so I can't say much more about it, but I have
> >>>> no doubts about it's ability to handle "complex situations", where
> >>>> multiple nodes / networks are involved.
> >>>>
> >>>>> That script works for sure, within very limited cases imho
> >>>>>
> >>>>>>> kaboom, really ugly kaboom. thats what is very likely to happen
> >>>>>>> sooner
> >>>>>>> or later especially when it comes to homegrown automatism
> solutions.
> >>>>>>> even the commercial parts where much more time/work goes into such
> >>>>>>> solutions fail in a regular manner
> >>>>>>>
> >>>>>>>> The advantage of ZFS send/receive of datasets is, however, that
> >>>>>>>> you can consider it
> >>>>>>>> essentially atomic. A transport corruption should not cause
> >>>>>>>> trouble (apart from a failed
> >>>>>>>> "zfs receive") and with snapshot retention you can even roll
> >>>>>>>> back. You can’t roll back
> >>>>>>>> zpool replications :)
> >>>>>>>>
> >>>>>>>> ZFS receive does a lot of sanity checks as well. As long as your
> >>>>>>>> zfs receive doesn’t involve a rollback
> >>>>>>>> to the latest snapshot, it won’t destroy anything by mistake.
> >>>>>>>> Just make sure that your replica datasets
> >>>>>>>> aren’t mounted and zfs receive won’t complain.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Cheers,
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Borja.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> _______________________________________________
> >>>>>>>> freebsd-fs at freebsd.org <mailto:freebsd-fs at freebsd.org> mailing
> list
> >>>>>>>> https://lists.freebsd.org/mailman/listinfo/freebsd-fs
> >>>>>>>> To unsubscribe, send any mail to
> >>>>>>>> "freebsd-fs-unsubscribe at freebsd.org
> >>>>>>>> <mailto:freebsd-fs-unsubscribe at freebsd.org>"
> >>>>>>>>
> >>>>>>> _______________________________________________
> >>>>>>> freebsd-fs at freebsd.org <mailto:freebsd-fs at freebsd.org> mailing
> list
> >>>>>>> https://lists.freebsd.org/mailman/listinfo/freebsd-fs
> >>>>>>> To unsubscribe, send any mail to
> >>>>>>> "freebsd-fs-unsubscribe at freebsd.org
> >>>>>>> <mailto:freebsd-fs-unsubscribe at freebsd.org>"
> >>>
> >>> _______________________________________________
> >>> freebsd-fs at freebsd.org <mailto:freebsd-fs at freebsd.org> mailing list
> >>> https://lists.freebsd.org/mailman/listinfo/freebsd-fs
> >>> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org
> >>> <mailto:freebsd-fs-unsubscribe at freebsd.org>"
> >
> > _______________________________________________
> > freebsd-fs at freebsd.org mailing list
> > https://lists.freebsd.org/mailman/listinfo/freebsd-fs
> > To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"
> _______________________________________________
> freebsd-fs at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"
>

More information about the freebsd-fs mailing list