continuous backup solution for FreeBSD

[Zaphod Beeblebrox]

On Tue, Oct 7, 2008 at 4:13 AM, Evren Yurtesen <yurtesen at ispro.net> wrote:

>
> Zaphod Beeblebrox wrote:
>
>
>>  From my reading, Hammer is much more than a filesystem, but then you
>> probably havn't read about it yet.  By my reading, Hammer hits all their
>> feature points and does it better _because_ it's a filesystem.
>>
>
> I glanced through these actually:
> http://www.dragonflybsd.org/hammer/
> I didnt see anywhere that it will replace backup programs?
>

You need to read the PDF on that page.  On the first page of the PDF the 4th
point is "queue-less incremental mirroring"  If you read the PDF to
determine what this phrase means, you'll find it describes the filesystem as
a database indexed by a B+ tree of radix 64.  It mentions that you can
easily select all changes newer than a certain time.  The problem with
backup solutions that run on raw blocks is that you need meta-data to queue
up the blocks that have changed faster than you can ship them out.  Hammer
solves this by allowing you to select changes based on when you shipped out
the last batch of changes.  The granularity of this will be dependent on how
fast you can do this.


>
>  It's relatively simple.  Database replication solves the data backup
>> problem (I don't store application data outside the database).  Database
>> replication for both MySQL and PostgreSQL is relatively straight forward.
>>  As for the configuration of code and servers --- that is taken care of with
>> configuration management (it's really a bigger issue than just backing up a
>> filesystem) and installing a new server to take a place in the cluster is a
>> straight forward checkout from the CM system.  For things I really care
>> about staying up, add VRRP and an application design that is fault tolerant.
>>
>
> Have you ever tried to restore data from MySQL replication logs? :) Even if
> you use binary logging, when you want to go back in time. You will need to
> first restore the whole database first from normal backups then replay the
> logs until the point that you wanna be. There is no simple way to go back in
> time. That is of course you have backups. If you dont have backups because
> you think replication is a backup solution, you would be screwed. Totally
> more complicated that clicking from the web to select data and time and
> table and restore!


Largely why I use PostgreSQL.


> Also, you are thinking about a very small sized system. While replication
> might work if you are relatively small sized company (like 1-2 servers). If
> you have many independent servers with different databases inside you just
> cant use it. Even if you could replicate multiple boxes into one box, there
> would eventually be problems such as same named databases etc. and even
> then, you cant just easily restore the data if the user deletes all his data
> in his tables.


Again, this is application design.  I know of rather large organizations
that use PostgreSQL replication to deliver very serious and onerous SLAs.
Several of them have sponsored the Slovney (sp? russian for elephan --- the
PostgreSQL multi-master replication thing).


> Also that is not practical for users at all. For example I cant give an
> option to the user to restore his data by himself. While that is possible
> with most backup software easily.


Well... if we're talking about a random samba servers for windoze users, ZFS
is the tech you want.  Users can retrieve their own snapshots, etc.  ZFS was
designed with generalized fileserver duties in mind.


> And how do you propose that I restore a table in the database to of 1h
> before status? like you can do with a data backup solution? You are talking
> about a spare server solution not backup solution. Replication IS NOT
> backup. If you look at articles and information about database replication,
> almost all mention that it DOES NOT replace backups.
>

I use differential backups to give old database restore functionality.  I
use replication to give me a live alternative database for failures
(different risks, different solutions).


>  Well... no.  Backup software and strategies are just one available tool
>> for risk mitigation.  To know what tools you require, you must define your
>> risks.  Then with your list of risks you look at the cost of each tool and
>> find the toolset that suits you.  By the responses in this thread, it seems
>> like the set of FreeBSD developers and the set of people who desire this
>> solution are disjoint.
>>
>
> Right, I just cant use the tool I require. There is no way to take near
> continuous backups of FreeBSD filesystems.
>

You're free to pay anyone you'd like to implement this solution.  I'd be
happy to work for hire.  I can provide a quote if you're serious about it.
But if I was doing it for free, I'd want it to be fun and interesting.  I'd
be imlementing a server as well as a client.  In fact, if I were doing this
as my own open source project, I'd be looking at a better way of achieving
it.  FreeBSD is, after all, about doing things right.  But --- as I said ---
regardless of what I'd do if I were doing it for free, I will achieve the
goals set out for me if I'm paid.