CTM index on servers

[Stephen Montgomery-Smith]

On 01/28/2013 01:29 PM, Isaac (.ike) Levy wrote:
> Hi Stephen, ctm-users,
> 
> New to this list, coming from a lively thread on freebsd-stable@
> 
> For me and a lot of admins/users, fetching sources just got a lot harder for a *lot* of admins/users.  A page was raised to try to help cull facts out of all the confusing details, here:
> 
>   https://wiki.freebsd.org/UsersFetchingSource
> 
> At this point the page may be more confusing itself :)
> 
> --
> Enter: CTM
> 
> I'm delighted to learn about ctm(1) usage, and quite interested in using it to fetch REL and RELENG sources, and keep it up to date (to redistribute internally across a cluster of servers).
> 
> I know a *ton* of *BSD admins who love c[v]sup, but I don't know any who use ctm(1) any more, so I'm sorry if my questions are naive.
> 
> On Jan 28, 2013, at 1:53 PM, Stephen Montgomery-Smith wrote:
>>> On the FTP server, is there an index of which CTM delta files are on the FTP server?  I'd like to automate fetching just the new deltas...
>>>
>>> If there was perhaps 1 file with a consistent name, I could fetch that on a nightly basis and fetch the other CTM files.
>>
>> I can easily put in something like this.  Maybe the output of ls in each directory?
>>
>> If you have more ideas on this, sign up to the ctm-users mailing list, and we can discuss exactly what is wanted.
> 
> I'm very interested in seeing some kind of 'index' file put in place, a simple file which lists the ctm delta (and xEmpty) files.
> 
> This way, users could do something in a shell/cron job, like:
> 
> fetch ftp://ftp.freebsd.org/pub/FreeBSD/development/CTM/src-9/index
> #
> # diff against the last ctm file, or a directory listing containing the delta files
> # download/apply any ctm delta files which have not yet been applied
> 
> --
> It's an open question weather or not the entire CTM dir should have the index, or the individual releases (or both?)
> 
> e.g.
> /pub/FreeBSD/development/CTM/src-9/index
>   - vs -
> /pub/FreeBSD/development/CTM/index
> 
> (I assume it would be appropriate also to provide sig files for the index, as well.)
> 
> --
> The index could be as simple as:
> 
> $ echo "`date +%s`" ; find . -type f -name 'src*' | sort | sed 's/.*\///'
> 1359382951
> src-9.0400xEmpty.gz
> src-9.0401.gz
> src-9.0401.gz.sig
> src-9.0402.gz
> src-9.0402.gz.sig
> src-9.0403.gz
> src-9.0403.gz.sig
> $ 
> 
> Could get fancier as necessary later, (perhaps date modified for each file, etc…)
> 
> --
> With some sort of index, the ease-of-use functionality of c[v]sup could be trivially replicated in sh(1).
> Even just as a prototype, this is a task I will enthusiastically do, even if only for my own use- (I started something already which culls FTP tarballs, a less than ideal solution).
> 
> What do yall' think?

The index idea would be very easy for me to implement.  Unless anyone
has a serious objection, I will probably go ahead and implement it as
above.  Except I will not include the time stamp in the file, as the
file already has a modification time on it.  I will probably sign the
index file as well, but I don't think it is necessary, as if that
information is corrupted it is unlikely to do any damage.



But ... I use a script like this:

I store all the ctm files in /usr/home/stephen/ctm or subdirectories
thereof.  This script sees what is already there, and then tries to
fetch any future ctm deltas.


#!/bin/sh

cd /usr/home/stephen/ctm/incoming

bases=`find /usr/home/stephen/ctm -type f |
       sed -n -E -e 's+.*/++' -e 's/\.[[:digit:]]{4}\.gz//p' | sort -u`

for b in $bases; do
  while true; do
    index=`find /usr/home/stephen/ctm -type f -name "$b*" |
           sed -n -E 's/.*([[:digit:]]{4}).*/\1/p' |
           sort -n | tail -1 | xargs expr 1 + | xargs printf "%04d"`
    fetch ftp://ftp.freebsd.org/pub/CTM/$b/$b.$index.gz || break
  done
done

echo
ls