continuous backup solution for freebsd?
koitsu at FreeBSD.org
Mon Oct 6 17:59:44 UTC 2008
On Mon, Oct 06, 2008 at 08:07:30PM +0300, Evren Yurtesen wrote:
> First of all, I am not an r1soft advocate, but they seem to be making a
> software which is popular and affordable and interested in giving
> FreeBSD support... r1soft is not the issue here, the problem is that
> there is no way to do near continuous backups on FreeBSD servers.
> Jeremy Chadwick wrote:
>> That said, I'd like to know exactly how "low-level" R1Soft's software
>> truly is. dump(8), AFAIK, is "block-level" -- and that's a userland
>> program. Does R1Soft's software *truly* require kernel-land? I have
>> more to say on that issue (not against R1Soft, but speaking with regards
>> to the current state of FreeBSD's developer count) if it truly does.
> I think you might not have understood the concept of near continuous
> backups. The R1Soft backup monitors the filesystem operations and backs
> up written blocks. So it has to know what is written and when to be able
> to back it up. The dump command simply reads/writes the blocks. It cant
> only read changed blocks. It has to read the whole thing (inefficient).
It depends on how it's implemented, but in general, yes, I guess this
would advocate reliance on GEOM, which would be kernel. The thing is,
the GEOM gate class could be extended to handle this situation -- it's
a class intended for filesystem replication in real-time, over a
That said, I shall unleash with the comments I had originally planned on
including, but removed them since I felt it might be too hasty of me.
The sad reality with FreeBSD is that we do not have enough clueful folks
who are familiar with the kernel innards. Those who are clueful are
very busy (with other things, and with real life), and often do not have
the time to give direct/constant focus on a single item for long periods
of time. I have a mental list of those who are absolutely incredible
FreeBSD developers (and I will name one of them: pjd@, who should be
given tens of thousands of dollars, IMHO, for his work on bringing ZFS
to FreeBSD), but the list is small compared to how many *users* we have.
The learning curve for getting familiar with pieces of the FreeBSD
kernel is astoundingly large. I myself have tried it on a couple of
occasions, but lack of concise and up-to-date documentation makes it
very difficult to accomplish. (I'm familiar with very old operating
systems, such as MS-DOS, ProDOS, and GS/OS on the Apple IIGS -- FreeBSD
is far from those). Books are also not of much help, as I've been told
that the existing book which covers FreeBSD engineering models is "long
outdated" and that "many pieces now are completely different". A
complete and total moral killer right there. The book is for FreeBSD
5.2, by the way.
We cannot rely on the FreeBSD Documentation folks to write the necessary
docs either, because they do not have the knowledge of the kernel to
write such. As someone who's written software, I can assure you that
the only way to get good documentation for low-level pieces is to write
the documentation in parallel to the code; otherwise, you end up with
lots of "after-the-fact" reverse engineering efforts, which takes tons
of time, and requires a lot of communication between the code author(s)
and the documenters. We're talking thousands of hours here.
Requiring the user/developer to reverse-engineer hundreds of thousands
of lines of C code is not reasonable/plausible; hardly anyone is
willing to do that for free.
This is why Linux often has the upper hand: they have multiple eyes and
individuals fully familiar with different pieces of the kernel. If one
or two people go on hiatus or disappear (death, life, whatever), the
existing kernel piece does not sit in limbo for years -- there are other
people to pick up the responsibilities. Much of the FreeBSD kernel and
device layer does not have this degree of freedom; much is
single-person, single-maintainer, single-point-of-failure.
Then there's commercial company support -- by that I mean, actual
hardware vendors that support the OS. FreeBSD has some of this, but
most are very small companies (few employees), with limited funds,
or have very *very* limited/specific focus; there are a couple big
ones, but they are few and far between. Linux has hundreds, and many
of the vendors are *very* large. In fact, the support is so large
that freelance Linux developers are able to get things like development
PCI boards for new NICs from the vendors directly; FreeBSD? Rare. What
this means is that the commercial world takes Linux seriously, while
FreeBSD not so. Sorry, but that's reality.
It amazes me how "easy" someone can pick up programming something in
kernel-land for Linux, while for FreeBSD it just doesn't happen on a
regular basis. When I see it happen, it's bizarre -- suddenly out of no
where comes this one fellow (we'll call him Bob), appearing on a mailing
list with a bunch of patches. Heard of him before? Nope, but here he
is, and somehow he engineered all of this. What's his background? I
don't know, maybe some old guy who lives in a cave and has been studying
BSD code in the steam tunnels; who knows. It's like they literally come
out of the woodwork, while I don't see this sort of behaviour with
Linux. With Linux, it's often "Hi I work for <large vendor>, we're
adding support for Linux, I need some help with regards to the following
kernel piece..." and they've got responses from 20 people in 24 hours.
What I'm saying is that Linux has the upper hand here. More eyes, more
people, more developers, larger community, larger vendor support, and
much **much** faster turn-around time on fixes/bugs. We can sit here
and argue about those facts all we want (it's the equivalent of doing
burn-outs in an AMC Pacer in a parking lot -- wasted time, zero gain),
but nothing changes the facts.
>>> Continuous backups as well as bare-metal-restore seem to be a key
>>> feature for many hosters.
>> Regarding continuous backups: the GEOM gate class could be used for
>> this. Meaning, I think it could be used as an alternate to R1Soft's
> The GEOM gate allows mirroring to a remote machine, am I not right? That
> would be more or less same as same as using RAID. The continuous backup
> (or near continuous) means that you can restore the filesystem to a
> point like 15 minutes ago, or 1 hour ago.
What you're talking about sounds like filesystem snapshots, with an
*immense* amount of granularity. Enterprise-level filers have this
capability (I'm talking Network Appliance), and UFS2 with softupdates
have it (called snapshots; but please be aware that there are *HUGE*
problems with it, and it should not be relied upon for this kind of
functionality) -- but nothing that can be restored within *minutes*.
Even Netapp filers do not have that kind of granularity -- the amount of
disk it would require would be astounding. Netapp filers often do
snapshot generation hourly or nightly (it's configurable how often);
minutes is unheard of.
ZFS also has snapshot capability, but does not have real-time filesystem
mirroring capabilities over a network (keyword: real-time).
> Besides, I hear geom might
> have network delay problems and it is much more complicated setup to
> build two machines in mirror configuration just for backup purposes as
> well as you cant restore to a point in the past.
Well, GEOM gate is the only thing I know of which replicates filesystems
over a network in real-time.
>> Regarding bare-metal restoration I'm not aware of how to do that under
>> FreeBSD, Linux, or even Solaris "with ease". In most cases, companies
>> develop their own PXE-booting environments which wipe the disks and
>> reinstall + restore data as they see fit. There is no "standard".
> OK. Actually there is more than one solution which can do
> bare-metal-restores for FreeBSD also. However those solutions at best
> rely on nightly backups of the filesystems. With R1Soft, you can restore
> the system to only few minutes before the total meltdown.
> Unrelated to bare metal restore, with normal backups you are not taking
> backups of files which are created/deleted often. For example this can
> be customer mails or if a hacker hacks the box and removes his trails.
> Even sometimes customers upload some file and remove from their computer
> the same they and then accidentally remove from the server. With R1Soft
> backup the data would go into the backup server right away and you an
> restore every single file independent of when it was put or removed.
Right. We're definitely talking about snapshots, at least in concept.
The fact that you're able to restore data within *minutes* is pretty
impressive. I'm curious what sort of disk requirements are needed
though (I guess it depends on how often changes happen on the
>>> FreeBSD is loosing users because of this issue.
>> Why does the "number of FreeBSD users" matter? Quantity does not
>> necessarily represent quality.
> Thats a perfectly fine statement. But a quality product would be nothing
> without users. As well as this problem effects the quality. Consider a
> system which has sensitive data which shouldnt get lost, with continuous
> data protecton you can restore such failed system to only few minutes
> before the failure point. Doing this is currently impossible with
> FreeBSD. Best we can do is to return to previous snapshot taken (which
> might be a day old). This is an important design criteria since
> restoring the lost data might be time consuming and expensive. Thge
> issue is not even r1soft, they are just the most popular company giving
> such solution, only if there was at least one backup solution which
> could provide near continuous data protection...
Part of the "design issue" here is that there's two concepts being
merged into one thing: snapshots and backup restoration.
I for one have never correlated snapshots and backup restorations
(bare-metal recovery). I consider them completely separate things, and
handled *very* differently. I have a feeling that no one's done this on
FreeBSD because the amount of effort required is quite large. Someone
did mention HAMMER on DragonflyBSD, but I have no knowledge of it or
what it provides -- that said, Matt (Dillon)'s stuff is usually very,
> In addition to this, near continuous backups create less load on boxes
> with a lot of reads but little writes. Standart backups have to scan all
> the files to detect which files were changed.
It depends on how the filesystem is done. For example, with UFS2+SU
snapshots, snapshot generation can take literally hours: completely
unreasonable. While with ZFS, snapshot generation usually takes 2-3
seconds -- even on massive changes (e.g. take a snapshot, then rm a
600MB ISO image, then compare present vs. snapshot -- the diff is
something like 40KBytes).
>> I'm sorry for sounding anti-FreeBSD, but the reality is that people
>> should use whatever solutions work best for them -- if that's using
>> Windows, Solaris, or Linux, great! Remember that open-source is about
>> choice: and choice means supporting the possibility that someone chooses
>> something else. Blind one-sided advocacy is very damaging to the
>> open-source model and concept.
> I agree, and please dont shoot the messenger :) I just have a bunch of
> customers who would use FreeBSD but not using only because of this
> problem. In addition to that I myself would like to use near continuous
> backups as well.
Understood. I now realise the full importance of what it is you've
described, and what R1Soft has developed. Thank you for taking the time
to educate me -- I appreciate it!
> I was just trying to inform the FreeBSD community here so if somebody
> can have some time to divert to giving the right advices to r1soft then
> we all could benefit from it. It doesnt even have to be free even, with
> a reasonable price they can probably hire somebody to work for building
> the basics of this feature.
> So the real question is, is there anybody who is willing and have the
> experience to help on this issue?
The response you're going to get is: "why don't you state how much
you're willing to pay for this feature, so that anyone who IS interested
can decide if that amount of money is worth the time required?"
My response is different: this sort of thing should definitely be pawned
off onto the FreeBSD Foundation. IMHO, this is the sort of thing the
Foundation *should* be handling. There is money there, and this sounds
like a project which could benefit FreeBSD as a whole. It's possible
that R1Soft, if paid, would take up such a challenge, assuming some key
folks (like Kirk McKusick; not volunteering him, just saying he has
experience with filesystems) could help with the development process and
learning curve. I can't speak for the Foundation, but it really sounds
like that's the way to go with this. I don't think you'll get any
responses from interested parties on freebsd-questions. :-)
| Jeremy Chadwick jdc at parodius.com |
| Parodius Networking http://www.parodius.com/ |
| UNIX Systems Administrator Mountain View, CA, USA |
| Making life hard for others since 1977. PGP: 4BD6C0CB |
More information about the freebsd-questions