continuous backup solution for freebsd?

[Evren Yurtesen]

Jeremy Chadwick wrote:

> What I'm saying is that Linux has the upper hand here.  More eyes, more
> people, more developers, larger community, larger vendor support, and
> much **much** faster turn-around time on fixes/bugs.  We can sit here
> and argue about those facts all we want (it's the equivalent of doing
> burn-outs in an AMC Pacer in a parking lot -- wasted time, zero gain),
> but nothing changes the facts.

Sorry, I had to remove the whole bunch of text that you wrote :) but I get the 
point.

I think it is a funny historical fact that BSD was commercially licensed way too 
long to allow Linux to be developed at first place. If BSD was not commercial at 
that times, Linus Torvalds probably wouldnt have started writing the Linux 
kernel. Thus we wouldnt be having this sort of conversation now and it might as 
well be that Microsoft wouldnt have become so huge. If we look at this from that 
point of view then eventually all BSD and Linux etc. are bound to disappear in 
time and Microsoft will stand all alone.

But things can change one step at a time. I prefer(or try) to look at this 
positively. I thought it wouldnt hurt to ask for help if somebody could contact 
r1soft and perhaps ask a pile of money to develop a driver. It would have been a 
win-win situation eh?

> Right.  We're definitely talking about snapshots, at least in concept.
> 
> The fact that you're able to restore data within *minutes* is pretty
> impressive.  I'm curious what sort of disk requirements are needed
> though (I guess it depends on how often changes happen on the
> filesystem).

Well it is not so fine grained (5 to 10 minutes intervals as mentioned).
http://www.r1soft.com/CDP.html
(there is more information in the link above, with links to outside sources on 
the concept such as wikipedia articles etc.)

I know some large hosters who use this technology with Linux servers. As a 
matter of fact the only reason they went with Linux instead of FreeBSD is 
because they cant get CDP with FreeBSD. I can ask how much space it is using and 
return back to you.

But if you think about it for a second, a traditional backup program would copy 
the whole file even if there was 1 byte changed in it. Lets say 10mbyte file and 
1 byte is changed. R1soft copies only 1 byte. Sure enough the tables can turn 
around if the filesystem was modified really a lot. But it looks like this type 
of solution is mostly effective (at least I didnt see anywhere that anybody is 
complaining that it is using too much disk space yet).

The best is, all it would take for FreeBSD users to be able to utilize this 
technology is a driver to interface with r1soft agent and buy a license. Now I 
am not expecting anybody to write this for free or nobody is obligated to help. 
I just dont know anybody who can help so I thought I would drop in a line here so...

> I for one have never correlated snapshots and backup restorations
> (bare-metal recovery).  I consider them completely separate things, and
> handled *very* differently.  I have a feeling that no one's done this on
> FreeBSD because the amount of effort required is quite large.  Someone
> did mention HAMMER on DragonflyBSD, but I have no knowledge of it or
> what it provides -- that said, Matt (Dillon)'s stuff is usually very,
> very good.

I also dont know much about HAMMER either. But it doesnt look like it will make 
mainstream usage anytime soon on FreeBSD if it ever does. Actually I found a 
nice document here:
http://www.dragonflybsd.org/hammer/hammer.pdf
http://www.dragonflybsd.org/hammer/index.shtml

> It depends on how the filesystem is done.  For example, with UFS2+SU
> snapshots, snapshot generation can take literally hours: completely
> unreasonable.  While with ZFS, snapshot generation usually takes 2-3
> seconds -- even on massive changes (e.g. take a snapshot, then rm a
> 600MB ISO image, then compare present vs. snapshot -- the diff is
> something like 40KBytes).

Yes, but r1soft backup can restore a single file at a consistent state without 
restoring the whole filesystem from a graphical user interface and can restore 
mysql databases at a table level. While I agree that there might be different 
solutions that I dont know about, it just takes a driver to get this 
functionality on current FreeBSD systems without everybody to change to ZFS or 
HAMMER. One has to think, would people change their filesystems or install a 
driver? :) I would rather pay license fee to a backup program and use the 
driver. The price of the software is very well justified if I can return back to 
5min before in my backups. The data I might loose is much more expensive.

>>> I'm sorry for sounding anti-FreeBSD, but the reality is that people
>>> should use whatever solutions work best for them -- if that's using
>>> Windows, Solaris, or Linux, great!  Remember that open-source is about
>>> choice: and choice means supporting the possibility that someone chooses
>>> something else.  Blind one-sided advocacy is very damaging to the
>>> open-source model and concept.
>> I agree, and please dont shoot the messenger :) I just have a bunch of  
>> customers who would use FreeBSD but not using only because of this  
>> problem. In addition to that I myself would like to use near continuous  
>> backups as well.
> 
> Understood.  I now realise the full importance of what it is you've
> described, and what R1Soft has developed.  Thank you for taking the time
> to educate me -- I appreciate it!

There are other companies with similar products for Linux ( you can search for 
continuous backup in google to find ) . However R1Soft is the most active one. 
Besides the possibility of continous backup, they have been partnering up with 
hosting control panel software vendors, for example they support cPanel or 
H-Sphere where the users can restore their own content from backups etc. Since 
most of the servers on the internet are used for such services, it makes much sense.

While there are other backup or backup like solutions ranging from HAMMER and 
ZFS built in features or Bacula etc. as far as I know these do not allow such 
flexibility.

>> I was just trying to inform the FreeBSD community here so if somebody  
>> can have some time to divert to giving the right advices to r1soft then  
>> we all could benefit from it. It doesnt even have to be free even, with  
>> a reasonable price they can probably hire somebody to work for building  
>> the basics of this feature.
>>
>> So the real question is, is there anybody who is willing and have the  
>> experience to help on this issue?
> 
> The response you're going to get is: "why don't you state how much
> you're willing to pay for this feature, so that anyone who IS interested
> can decide if that amount of money is worth the time required?"

It is a possibility but then I am not working for R1Soft. People who has the 
knowledge to be able to pull this off can ask this to R1Soft. I am probably 
reporting a job possibility to an idle developer if there is such thing as an 
idle developer.

R1soft is even willing to develop this themselves as far as I can tell. Maybe 
they just need the right information to be pointed out to them to start 
development only.

> My response is different: this sort of thing should definitely be pawned
> off onto the FreeBSD Foundation.  IMHO, this is the sort of thing the
> Foundation *should* be handling.  There is money there, and this sounds
> like a project which could benefit FreeBSD as a whole.  It's possible
> that R1Soft, if paid, would take up such a challenge, assuming some key
> folks (like Kirk McKusick; not volunteering him, just saying he has
> experience with filesystems) could help with the development process and
> learning curve.  I can't speak for the Foundation, but it really sounds
> like that's the way to go with this.  I don't think you'll get any
> responses from interested parties on freebsd-questions.  :-)
> 

Thanks, I like your response :) But I dont know anybody in FreeBSD Foundation or 
anybody who can help. That is exactly why I came to here to mention the issue. 
If somebody knows somebody who can help then please pass along the word. I am in 
no way saying that anybody is obligated to do anything. Just informing that this 
great technology exists but FreeBSD users are not able to use it yet because of 
development issues.

My personal opinion is that backups are an important issue for a serious company 
and it is a key selection criteria when choosing components and design for a 
system. I might be right or wrong of course. I just got pissed off enough that 
lately people I work with just wouldnt use FreeBSD only because of this 
problem... :( if you check r1soft forums, you can also see that I was bugging 
r1soft about this issue a lot and they ended up saying that it is probably not 
possible in FreeBSD to create such kernel module. At least I feel satisfied that 
I was doing or trying to do something in my skill levels to help to FreeBSD 
community and I am happy that some people are listening. But it is obvious that 
I dont know the right people to directly contact and inform about these stuff so 
please go ahead and forward this information to whomever you feel can help.

Thanks,
Evren