Getting distfiles for entire tree

[Vadim Goncharov]

Hello freebsd-ports,

I need to maintain a local distfiles mirror (all N-ty gigabytes) for
almost entire ports tree (some lang categories excluded). Also, I have
limited access to Internet - not all distfiles could be fetched (however
eventually they'll do). So, as I can't use plain ``cd /usr/ports && make
fetch'', I've made a script, which, for speeding up, fetches each category
in parallel.
Script was simple, so it hasn't got all, mainly because of interactive
ports, etc. So I searched a little and, after reading some tips,
modified script - commented ``find'' and inserted ``daemon''. Here it
is:

#!/bin/sh

CATEGORIES="accessibility archivers astro audio benchmarks biology cad comms converters databases deskutils devel dns editors emulators finance ftp games graphics irc japanese java lang mail math mbone misc multimedia net net-im net-mgmt net-p2p news palm print russian science security shells sysutils textproc ukrainian www x11 x11-clocks x11-fm x11-fonts x11-servers x11-themes x11-toolkits x11-wm" 
LOG=/data/FreeBSD/fetchlog 

# vars for fetch 
export HTTP_TIMEOUT=30 
export FTP_TIMEOUT=30 

for i in $CATEGORIES ; do 
        cd /usr/ports/$i || exit 1 
#       find . -type d -mindepth 1 -maxdepth 1 -exec sh -c "cd {} && make fetch" \; > ${LOG}-$i 2>&1 & 
        daemon sh -c "BATCH=yes NO_IGNORE=yes PACKAGE_BUILDING=yes make -k checksum" > ${LOG}-$i 2>&1 
done

First version (find) forked 300 processes at beginning. Second (daemon)
forked 400, and, as it was going, I saw strange things, e.g. error
messages in logs like that:

===> abclock
Abort trap 
*** Error code 134 (continuing) 

or

===> xtu 
=> MD5 Checksum OK for xtu-1.2.tar.gz. 
=> SHA256 Checksum OK for xtu-1.2.tar.gz. 
*** Signal 6 (continuing) 
`checksum' not remade because of errors.

Why did it happen ? Lack of resources? But I ran it as root, limits were
soft, and at least that 400 processes forked!

But then some anothe shit happened - system began to build and install
packages without checking if they were actually installed, e.g. it made
mess installing expat2 over expat-1.95 (I had to deal with it manually
later), so I was forced to interrupt the script.

So, what happened (why it began to install, why errors?) and how should
I create local distfiles mirror ?

-- 
Best regards,
 Vadim Goncharov   ICQ UIN 166852181   mailto:vadim_nuclight at mail.ru