Regarding zfs send / receive

[Waitman Gobble]

Waitman Gobble
San Jose California USA
On Apr 4, 2013 2:07 PM, "Joar Jegleim" <joar.jegleim at gmail.com> wrote:
>
> Hi Terje !
> sorry for late reply, I've been checking my mail, forgetting that all my
> mailing list mail are sorted into their own folders skipping inbox :p
>
> the zfs sync setup is a huge advantage over rsync simply because
> incremental rsync of the volume takes ~12 hours, while the zfs
differential
> snapshot's usually take less than a minute . Though it's only ~1TB of
data,
> it's  more than 2 million jpegs which rsync have to stat ...
> I'm guessing my predecessor who chose this setup, over for instance HAST,
> didn't feel confident enough regarding HAST in production ( I'm looking
> into that for a future solution) .
>
> There's no legacy stuff on the receiving end, old pools are deleted for
> every sync. I haven't got my script here but google pointed me too
> https://github.com/hoopty/zfs-sync/blob/master/zfs-sync which look like a
> script very similar to the one I'm using .
> In fact, I'm gonna take a closer look at that script and see what differs
> from my script (apart from it being much prettier :p )
> I didn't know about zpool.cache, gonna check that tomorrow, thanks.
>
>
>
> --
> ----------------------
> Joar
> Jegleim
> Homepage: http://cosmicb.no
> Linkedin: http://no.linkedin.com/in/joarjegleim
> fb: http://www.facebook.com/joar.jegleim
> AKA: CosmicB @Freenode
>
> ----------------------
>
> On 2 April 2013 14:40, Terje Elde <terje at elde.net> wrote:
>
> > On 2. apr. 2013, at 13.44, Joar Jegleim wrote:
> > > So my question(s) to the list would be:
> > > In my setup have I taken the use case for zfs send / receive too far
> > > (?) as in, it's not meant for this kind of syncing and this often, so
> > > there's actually nothing 'wrong'.
> >
> > I'm not sure if you've taken it too far, but I'm not entirely sure if
> > you're getting any advantage over using rsync or similar for this kind
of
> > thing.
> >
> > First two things that spring to mind:
> >
> > Do you have any legacy stuff on the receiving machine?  Things like
> > physically removed old zpools, that are still in zpool.cache, seems to
slow
> > down various operations, including creation of new stuffs (such as the
> > snapshots you receive).
> >
> > Also, you don't mention if you're deleting old snapshots on the
receiving
> > end?  If you're doing an incremental run every 15 minutes, that's
something
> > like 3000 snapshots pr. month, pr. filesystem.
> >
> > Terje
> >
> >
>

hi,
i have a similar situation. its better to only rsync new stuff in this
case, because you should know when somebody ads something new.

for example, a user uploads 200 new images, these are marked 'to sync' and
are transferred to the other servers. letting rsync figure out what's new
just isnt practical.

an idea, works for me. hope it helps.

Waitman Gobble
San Jose California _______________________________________________
> freebsd-questions at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-questions
> To unsubscribe, send any mail to "
freebsd-questions-unsubscribe at freebsd.org"