CTM index on servers

[Isaac (.ike) Levy]

Hi Stephen, ctm-users,

New to this list, coming from a lively thread on freebsd-stable@

For me and a lot of admins/users, fetching sources just got a lot harder for a *lot* of admins/users.  A page was raised to try to help cull facts out of all the confusing details, here:

  https://wiki.freebsd.org/UsersFetchingSource

At this point the page may be more confusing itself :)

--
Enter: CTM

I'm delighted to learn about ctm(1) usage, and quite interested in using it to fetch REL and RELENG sources, and keep it up to date (to redistribute internally across a cluster of servers).

I know a *ton* of *BSD admins who love c[v]sup, but I don't know any who use ctm(1) any more, so I'm sorry if my questions are naive.

On Jan 28, 2013, at 1:53 PM, Stephen Montgomery-Smith wrote:
>> On the FTP server, is there an index of which CTM delta files are on the FTP server?  I'd like to automate fetching just the new deltas...
>> 
>> If there was perhaps 1 file with a consistent name, I could fetch that on a nightly basis and fetch the other CTM files.
> 
> I can easily put in something like this.  Maybe the output of ls in each directory?
> 
> If you have more ideas on this, sign up to the ctm-users mailing list, and we can discuss exactly what is wanted.

I'm very interested in seeing some kind of 'index' file put in place, a simple file which lists the ctm delta (and xEmpty) files.

This way, users could do something in a shell/cron job, like:

fetch ftp://ftp.freebsd.org/pub/FreeBSD/development/CTM/src-9/index
#
# diff against the last ctm file, or a directory listing containing the delta files
# download/apply any ctm delta files which have not yet been applied

--
It's an open question weather or not the entire CTM dir should have the index, or the individual releases (or both?)

e.g.
/pub/FreeBSD/development/CTM/src-9/index
  - vs -
/pub/FreeBSD/development/CTM/index

(I assume it would be appropriate also to provide sig files for the index, as well.)

--
The index could be as simple as:

$ echo "`date +%s`" ; find . -type f -name 'src*' | sort | sed 's/.*\///'
1359382951
src-9.0400xEmpty.gz
src-9.0401.gz
src-9.0401.gz.sig
src-9.0402.gz
src-9.0402.gz.sig
src-9.0403.gz
src-9.0403.gz.sig
$ 

Could get fancier as necessary later, (perhaps date modified for each file, etc…)

--
With some sort of index, the ease-of-use functionality of c[v]sup could be trivially replicated in sh(1).
Even just as a prototype, this is a task I will enthusiastically do, even if only for my own use- (I started something already which culls FTP tarballs, a less than ideal solution).

What do yall' think?

Best,
.ike