Backup solution suggestions [ggated]

[Johan Ström]

On Jan 17, 2008, at 09:30 , Ulrich Spoerlein wrote:

> On Jan 17, 2008 1:31 AM, Johan Ström <johan at stromnet.se> wrote:
>>> Export the disk on the backup server with ggated. Bind it on the
>>> client
>>> with ggatec. Slap a GELI or GBDE encryption on top of it and then
>>> put a
>>> ZFS on top of it.
>>>
>>> You can mount/import this "remote" ZFS at will and do your zfs
>>> send/receive on your local box. Nothing ever leaves your box
>>> unencrypted.
>>
>> Now that is a cool solution! That actually sounds like something  
>> doable.
>> I tried it out some at home between a 6.2 box (client) and 7.0 box
>> (server), hosting the system in a ZFS "sparse volume" with a
>> predefined size, exported that via ggated and connected ggatec on the
>> client box. I then did some experimentation with just newfs, and it
>> worked great!
>> The only downside with this would be that the size is fixed. So I
>> played around a bit with setting the volsize property in ZFS and it
>> seemd to work just fine. zfs list reported the new, bigger, size.
>> Restarted ggatec and did a growfs, and then remounted.. Yay bigger
>> disk :)
>> Then I went on do do some geli test, geli'ed /dev/ggate0 and
>> newfs'ed, mounted and played around a bit. All fine.. Now came the
>> problem, i unmounetd it, expanded the zfs volume a bit more,
>> restarted ggatec and tried to attach it using geli again (note, I
>> have no idea if this is supposed to work at all, I'm just testing.
>> Havent read such things anywhere). Now I got Invalid argument.
>> Im not realy sure about how GEOM works, but if I recall correct it
>> uses the last sectors of the disk? If I moved X bytes of data from
>> old end of disk to new end of disk, would that make GELI work? If I
>> can get that to work, then this would be a kickass solution (all
>> encryption stuff works great, I don't have to allocate all space
>> immediatly, I can expand it later without destroying data and
>> starting from scratch etc).
>
> I'm pretty certain that GELI cannot handle variable sized disks. But
> you could add GVIRSTOR into the mix. But I'd just allocate the
> necessary space and be done with it. Adding yet another layer is
> asking for trouble, imho.

Okay.

>
>> Some other questions, more related to ggated/c. Is this stable? Good
>> working? how does it handle failure situations? Anyone using it for
>> production systems?
>
> From my personal experience (which is rather limited): No, barely,  
> bad, hell no.
>
> There were/are some open PRs about ggate. I had troubles with
> gmirror+ggate in that it would deadlock every other hour on SMP
> systems (try removing option PREEMPTION if that bug hits you).

Your no,barely, bad hell no seems to fit pretty good.. I did some  
testing during the night with the above (non-production) setup.
What I did was doing some rsyncing over the night:

while true ; do
         echo "`date` Clearing vmail" >> logfile
         rm -rf vmail
         echo "`date` Starting rsync" >> logfile
         rsync -vr /usr/var/vmail . |tee -a logfile
         echo "`date` Rsync finished " >> logfile
done

I started this at ~02.0. The results? A freshly rebooted 6.2 (6.2- 
RELEASE-p6 FreeBSD 6.2-RELEASE-p6 #0: Fri Jul 27 15:47:50 UTC 2007)  
box in the morning..
Looking at the messages:

Jan 18 05:33:25 phomca kernel: GEOM_ELI: Crypto WRITE request failed  
(error=5). ggate0.eli[WRITE(offset=8844480512, length=4096)]
Jan 18 05:33:25 phomca kernel: GEOM_ELI: Crypto WRITE request failed  
(error=5). ggate0.eli[WRITE(offset=8844484608, length=4096)]
Jan 18 05:33:25 phomca kernel: GEOM_ELI: Crypto WRITE request failed  
(error=5). ggate0.eli[WRITE(offset=8844488704, length=4096)]
Jan 18 05:33:25 phomca kernel: GEOM_ELI: Crypto WRITE request failed  
(error=5). ggate0.eli[WRITE(offset=8844492800, length=4096)]
Jan 18 05:33:25 phomca kernel: GEOM_ELI: Crypto WRITE request failed  
(error=5). ggate0.eli[WRITE(offset=8844517376, length=4096)]
... more of the same...
Jan 18 05:33:25 phomca kernel: g_vfs_done():ggate0.eli[WRITE 
(offset=8844480512, length=4096)]error = 5
Jan 18 05:33:25 phomca kernel: GEOM_ELI: Crypto WRITE request failed  
(error=5). ggate0.eli[WRITE(offset=8844640256, length=32768)]
..more of the same...
Jan 18 05:33:25 phomca kernel: GEOM_ELI: Crypto WRITE request failed  
(error=5). ggate0.eli[WRITE(offset=8844988416, length=4096)]
Jan 18 05:33:25 phomca kernel: g_vfs_done():ggate0.eli[WRITE 
(offset=8844484608, length=4096)]error = 5
Jan 18 05:33:25 phomca kernel: g_vfs_done():ggate0.eli[WRITE 
(offset=8844488704, length=4096)]error = 5
Jan 18 05:33:25 phomca kernel: g_vfs_done():ggate0.eli[WRITE 
(offset=8844492800, length=4096)]error = 5
Jan 18 05:33:25 phomca kernel: g_vfs_done():ggate0.eli[WRITE 
(offset=8844517376, length=4096)]error = 5
Jan 18 05:33:25 phomca kernel: GEOM_ELI: Crypto WRITE request failed  
(error=5). ggate0.eli[WRITE(offset=8844992512, length=4096)]
...more of the same...
Jan 18 05:33:25 phomca kernel: GEOM_ELI: Crypto WRITE request failed  
(error=5). ggate0.eli[WRITE(offset=65536, length=4096)]
Jan 18 05:33:25 phomca kernel: g_vfs_done():ggate0.eli[WRITE 
(offset=8844521472, length=4096)]error = 5
Jan 18 05:33:25 phomca kernel: g_vfs_done():ggate0.eli[WRITE 
(offset=8844496896, length=4096)]error = 5
Jan 18 05:33:25 phomca kernel: g_vfs_done():ggate0.eli[WRITE 
(offset=8844500992, length=4096)]error = 5
Jan 18 05:33:25 phomca kernel: g_vfs_done():ggate0.eli[WRITE 
(offset=8844505088, length=4096)]error = 5
..and more of those...
Jan 18 05:33:25 phomca kernel: g_vfs_done():ggate0.eli[WRITE 
(offset=8845176832, length=4096)]error = 5
Jan 18 05:33:25 phomca kernel: g_vfs_done():ggate0.eli[WRITE 
(offset=5636096, length=4096)]error = 5
Jan 18 05:33:25 phomca kernel: g_vfs_done():ggate0.eli[WRITE 
(offset=65536, length=4096)]error = 5
Jan 18 05:33:43 phomca kernel: GEOM_ELI: Crypto WRITE request failed  
(error=5). ggate0.eli[WRITE(offset=5636096, length=4096)]
Jan 18 05:33:43 phomca kernel: GEOM_ELI: Crypto WRITE request failed  
(error=5). ggate0.eli[WRITE(offset=8127250432, length=16384)]
Jan 18 05:33:43 phomca kernel: GEOM_ELI: Crypto WRITE request failed  
(error=5). ggate0.eli[WRITE(offset=8480587776, length=16384)]
...more of those..
Jan 18 05:33:43 phomca kernel: g_vfs_done():ggate0.eli[WRITE 
(offset=8834072576, length=16384)]error = 5
Jan 18 05:35:42 phomca syslogd: kernel boot file is /boot/kernel/kernel

What I dont have is a coredump, judging from dmesg -a savecore wasnt  
even run.. running it now, 5 hours later, didnt find any cores.

The other end (7.0 server) wasnt affected at all.

Not realy sure what it had been doing, because looking at my  
bandwidth graphs from the switch, nothing was done at all.. It didnt  
even go through one iteration of rsync... ~7.5k files/directorys  
seems to have been transfered, then the log doesnt say more. But  
according to the BW graph, after ~03.00 no traffic was sent at all...

Some known bug with 6.2?

>> Yes this is for backup only so minor glitches
>> might be acceptable for me, but I'd rather know about those  
>> beforehand.
>
> Give it a shot, if your systems stay up and stable, good. If the link
> breaks from time to time, I think ZFS should be able to recover most
> of it. Since it is your backup, I'd try to break it in interessting
> ways first, to get a feel for how robust it is.
>
> Uli
>

So, crashing the backuped box is not a minor glitch. Thus this doesnt  
seem to be useable AT ALL :/ Too bad..

--
Johan