rsync and the ports tree

[Peter Kryszkiewicz]

Sounds like a well-thought out backup strategy. I've started to use your
methods here, and I'm building ports I need at the same time, but the
wireless here is not password protected so is incredibly slow as there are
lots of leaches on the system. But while I'm plodding along, I have a few
more questions:

On Wed, Oct 26, 2011 at 8:07 AM, C. P. Ghost <cpghost at cordula.ws> wrote:

> On Wed, Oct 26, 2011 at 12:39 PM, Peter Kryszkiewicz
> <tundra2bear at gmail.com> wrote:
> > I have several machines installed in my temporary location and only my
> > laptop gets the internet through wireless. So far I've been building
> ports
> > on the other machines by rsync'ing the distfiles from the laptop as I
> need
> > them (all machines have the same FreeBSD 8.2 installed).
> >
> > The problem comes after I did a 'portupgrade -a' on the laptop. To ensure
> > the other ports trees are in sync, can I rsync the /usr/ports directory
> to
> > the other machines? Since some of them are different architectures (amd64
> > multicore for instance) I ran into situations where the distfiles are
> > different (for gcc for example).
>
> First of all, rsync is working perfectly if you want to
> distribute /usr/ports/distfiles, /usr/ports to your internal
> machines, even when they are not of the same architecture.
> I'm doing this with a BIG farm of servers running i386, amd64,
> and sparc64 for a long, long time.
>
> You only need to make sure to rsync the *union* of your
> /usr/ports/distfiles directories, or else it won't work.
>
> Say, on amd64 you have
>  /usr/ports/distfiles/some-distfile-for-amd64-only.tar.bz2
> and on i386 you have
>  /usr/ports/distfiles/some-distfile-for-i386-only.tar.bz2
>
> Yes, that happens every now and then.
>
> So you have to rsync both ways, so that you end up with
>  /usr/ports/distfiles/some-distfile-for-amd64-only.tar.bz2
>  /usr/ports/distfiles/some-distfile-for-i386-only.tar.bz2
> on both i386 and amd64 machines.
>

I've done that, it works well especially for the architecture differences.
gcc requires an additional distfile for the amd64 build).

>
> The catch is: look out for rsync's --delete flag! When some
> port managers delete old/stale distfiles, they may also delete
> distfiles for the *other* arches because they (rightly) think
> they are not needed here... and when you then rsync with --delete,
> that would (wrongly) propagate such deletes to those arches,
> and you end up with missing distfiles on the targets.
>
> Since I have more than just two arches, I use a slightly different
> 2-layer workflow:
>
> 0. I have 3 servers that are allowed to fetch files from the outside:
>     i386-master, amd64-master, sparc64-master.
>   and a whole bunch of i386-slave-NNN, amd64-slave-NNN and
>   sparc64-slave-NNN machines that would duplicate from their
>   relative masters via rsync.
>
>   On all -master(s), I keep $DISTFILES outside of /usr/ports
>   (on /usr/local/distfiles, with a symbolic link in /usr/ports
>      /usr/ports/distfiles -> /usr/local/distfiles)
>
> Initial update of i386-master, as usual:
>
> 1. On i386-master, csup /usr/ports.
>   Run portmaster as usual to upgrade everything.
>   This may delete old stale distfiles and non-i386-distfiles.
>   This may fetch additional generic and i386-specific distfiles.
>
> Copy the new /usr/ports (without distfiles) to the other
> arch masters:
>
> 2. rsync -av --delete i386-master:/usr/ports to amd64-master
>   and sparc64-master. CAUTION: Use --delete is okay, but only
>   because distfiles are not under /usr/ports, so as not to nuke
>   non-i386-specific distfiles of the other arches.
>
> Copy i386-master's NEW distfiles to the other arch masters:
>
> 3. rsync -av i386-master:/usr/local/distfiles to amd64-master
>   and sparc64-master. BEWARE: Don't use --delete here!
>   Do this to copy new generic distfiles (and i386) from
>   the i386-master build to amd64-master and sparc64-master.
>
> Update amd64-master and sparc64-master's ports as usual:
>
> 4. On amd64-master, run portmaster as usual to upgrade everything.
>   This may delete old stale distfiles and non-amd64-distfiles.
>   This may fetch additional (generic and) amd64-specific-distfiles.
>
> 5. On sparc64-master, run portmaster as usual to upgrade everything.
>   This may delete old stale distfiles and non-sparc64-distfiles.
>   This may fetch additional (generic and) sparc64-specific-distfiles.
>
> At this point, i386-master, amd64-master and sparc64-master are
> fully updated, and their /usr/local/distfiles directories are up
> to date w.r.t. their specific architectures. Now, copy everything
> from the masters to the slaves:
>
> 6. On every i386-slave-NNN, rsync -av --delete:
>     /usr/ports, /usr/local (including /usr/local/distfiles),
>     /var/db/pkg, /var/db/ports
>   from i386-master.
>
> 7. On every amd64-slave-NNN, rsync -av --delete:
>     /usr/ports, /usr/local (including /usr/local/distfiles)
>     /var/db/pkg, /var/db/ports
>   from amd64-master.
>

I can see the need to sync /var/db/ports, but isn't  /var/db/pkg specific to
each machine?

Same with /usr/local/distfiles (as far as the need to sync across machines)
but /usr/local/ in general is also specific to each machine and is huge,
storage-wise. I put git repositories there, like the kdenlive and ffmpeg
 sources I like to follow on graphics machines, while other machines have
TeX and family installed.

>
> 8. On every sparc64-slave-NNN, rsync -av --delete:
>     /usr/ports, /usr/local (including /usr/local/distfiles)
>     /var/db/pkg, /var/db/ports
>   from sparc64-master.
>
> You may also need to update entries in /etc and /usr/local/etc
> on the slaves.
>
> > If not rsync, what is the best way to keep multiple ports trees on
> different
> > hardware in sync, assuming everything runs FreeBSD 8.2?
> >
> > regards,
> > Peter Kryszkiewicz
>
> -cpghost.
>
> --
> Cordula's Web. http://www.cordula.ws/
>