rsync and the ports tree

C. P. Ghost cpghost at
Wed Oct 26 15:07:54 UTC 2011

On Wed, Oct 26, 2011 at 12:39 PM, Peter Kryszkiewicz
<tundra2bear at> wrote:
> I have several machines installed in my temporary location and only my
> laptop gets the internet through wireless. So far I've been building ports
> on the other machines by rsync'ing the distfiles from the laptop as I need
> them (all machines have the same FreeBSD 8.2 installed).
> The problem comes after I did a 'portupgrade -a' on the laptop. To ensure
> the other ports trees are in sync, can I rsync the /usr/ports directory to
> the other machines? Since some of them are different architectures (amd64
> multicore for instance) I ran into situations where the distfiles are
> different (for gcc for example).

First of all, rsync is working perfectly if you want to
distribute /usr/ports/distfiles, /usr/ports to your internal
machines, even when they are not of the same architecture.
I'm doing this with a BIG farm of servers running i386, amd64,
and sparc64 for a long, long time.

You only need to make sure to rsync the *union* of your
/usr/ports/distfiles directories, or else it won't work.

Say, on amd64 you have
and on i386 you have

Yes, that happens every now and then.

So you have to rsync both ways, so that you end up with
on both i386 and amd64 machines.

The catch is: look out for rsync's --delete flag! When some
port managers delete old/stale distfiles, they may also delete
distfiles for the *other* arches because they (rightly) think
they are not needed here... and when you then rsync with --delete,
that would (wrongly) propagate such deletes to those arches,
and you end up with missing distfiles on the targets.

Since I have more than just two arches, I use a slightly different
2-layer workflow:

0. I have 3 servers that are allowed to fetch files from the outside:
     i386-master, amd64-master, sparc64-master.
   and a whole bunch of i386-slave-NNN, amd64-slave-NNN and
   sparc64-slave-NNN machines that would duplicate from their
   relative masters via rsync.

   On all -master(s), I keep $DISTFILES outside of /usr/ports
   (on /usr/local/distfiles, with a symbolic link in /usr/ports
      /usr/ports/distfiles -> /usr/local/distfiles)

Initial update of i386-master, as usual:

1. On i386-master, csup /usr/ports.
   Run portmaster as usual to upgrade everything.
   This may delete old stale distfiles and non-i386-distfiles.
   This may fetch additional generic and i386-specific distfiles.

Copy the new /usr/ports (without distfiles) to the other
arch masters:

2. rsync -av --delete i386-master:/usr/ports to amd64-master
   and sparc64-master. CAUTION: Use --delete is okay, but only
   because distfiles are not under /usr/ports, so as not to nuke
   non-i386-specific distfiles of the other arches.

Copy i386-master's NEW distfiles to the other arch masters:

3. rsync -av i386-master:/usr/local/distfiles to amd64-master
   and sparc64-master. BEWARE: Don't use --delete here!
   Do this to copy new generic distfiles (and i386) from
   the i386-master build to amd64-master and sparc64-master.

Update amd64-master and sparc64-master's ports as usual:

4. On amd64-master, run portmaster as usual to upgrade everything.
   This may delete old stale distfiles and non-amd64-distfiles.
   This may fetch additional (generic and) amd64-specific-distfiles.

5. On sparc64-master, run portmaster as usual to upgrade everything.
   This may delete old stale distfiles and non-sparc64-distfiles.
   This may fetch additional (generic and) sparc64-specific-distfiles.

At this point, i386-master, amd64-master and sparc64-master are
fully updated, and their /usr/local/distfiles directories are up
to date w.r.t. their specific architectures. Now, copy everything
from the masters to the slaves:

6. On every i386-slave-NNN, rsync -av --delete:
     /usr/ports, /usr/local (including /usr/local/distfiles),
     /var/db/pkg, /var/db/ports
   from i386-master.

7. On every amd64-slave-NNN, rsync -av --delete:
     /usr/ports, /usr/local (including /usr/local/distfiles)
     /var/db/pkg, /var/db/ports
   from amd64-master.

8. On every sparc64-slave-NNN, rsync -av --delete:
     /usr/ports, /usr/local (including /usr/local/distfiles)
     /var/db/pkg, /var/db/ports
   from sparc64-master.

You may also need to update entries in /etc and /usr/local/etc
on the slaves.

> If not rsync, what is the best way to keep multiple ports trees on different
> hardware in sync, assuming everything runs FreeBSD 8.2?
> regards,
> Peter Kryszkiewicz


Cordula's Web.

More information about the freebsd-questions mailing list