Getting distfiles for entire tree
Vadim Goncharov
vadim_nuclight at mail.ru
Thu Mar 16 20:03:55 UTC 2006
Hello freebsd-ports,
I need to maintain a local distfiles mirror (all N-ty gigabytes) for
almost entire ports tree (some lang categories excluded). Also, I have
limited access to Internet - not all distfiles could be fetched (however
eventually they'll do). So, as I can't use plain ``cd /usr/ports && make
fetch'', I've made a script, which, for speeding up, fetches each category
in parallel.
Script was simple, so it hasn't got all, mainly because of interactive
ports, etc. So I searched a little and, after reading some tips,
modified script - commented ``find'' and inserted ``daemon''. Here it
is:
#!/bin/sh
CATEGORIES="accessibility archivers astro audio benchmarks biology cad comms converters databases deskutils devel dns editors emulators finance ftp games graphics irc japanese java lang mail math mbone misc multimedia net net-im net-mgmt net-p2p news palm print russian science security shells sysutils textproc ukrainian www x11 x11-clocks x11-fm x11-fonts x11-servers x11-themes x11-toolkits x11-wm"
LOG=/data/FreeBSD/fetchlog
# vars for fetch
export HTTP_TIMEOUT=30
export FTP_TIMEOUT=30
for i in $CATEGORIES ; do
cd /usr/ports/$i || exit 1
# find . -type d -mindepth 1 -maxdepth 1 -exec sh -c "cd {} && make fetch" \; > ${LOG}-$i 2>&1 &
daemon sh -c "BATCH=yes NO_IGNORE=yes PACKAGE_BUILDING=yes make -k checksum" > ${LOG}-$i 2>&1
done
First version (find) forked 300 processes at beginning. Second (daemon)
forked 400, and, as it was going, I saw strange things, e.g. error
messages in logs like that:
===> abclock
Abort trap
*** Error code 134 (continuing)
or
===> xtu
=> MD5 Checksum OK for xtu-1.2.tar.gz.
=> SHA256 Checksum OK for xtu-1.2.tar.gz.
*** Signal 6 (continuing)
`checksum' not remade because of errors.
Why did it happen ? Lack of resources? But I ran it as root, limits were
soft, and at least that 400 processes forked!
But then some anothe shit happened - system began to build and install
packages without checking if they were actually installed, e.g. it made
mess installing expat2 over expat-1.95 (I had to deal with it manually
later), so I was forced to interrupt the script.
So, what happened (why it began to install, why errors?) and how should
I create local distfiles mirror ?
--
Best regards,
Vadim Goncharov ICQ UIN 166852181 mailto:vadim_nuclight at mail.ru
More information about the freebsd-ports
mailing list