A new and better way to do "make readmes"?

[Jason Helfman]

On Thu, Feb 02, 2012 at 03:21:37PM -0600, Conrad J. Sabatier thus spake:
>[ Sorry to be so late in following up on this; lost track for a while ]
>
>On Sat, 28 Jan 2012 16:53:17 +0000
>Matthew Seaman <m.seaman at infracaninophile.co.uk> wrote:
>
>> On 28/01/2012 16:28, Conrad J. Sabatier wrote:
>> > <rubbing eyes in disbelief>  Am I understanding you correctly?  Are
>> > you saying you built 20,000+ port READMEs in only 9 seconds?!  How
>> > is that possible?  Or do you mean 9 seconds for each one?
>>
>> 9 seconds sounds quite reasonable for generating 23000 or so files.
>
>It sounds incredible to me!  :-)
>
>> >> > Selective updating isn't going to help because 99.9% of the time
>> >> > is spent in the categories and it only takes a single port
>> >> > update to make a category file obsolete.
>>
>> > This is the part I find troubling.  It would seem that it should be
>> > more work to create an individual port README, with its plucking the
>> > appropriate line out of the INDEX-* file and then parsing it into
>> > its respective pieces and filling in a template, than to simply
>> > string together a list of references to a bunch of already built
>> > port READMEs into a category README.
>> >
>> > What am I not getting here?
>>
>> No -- you're quite right.  You could generate the category README.html
>> files entirely from the data in the INDEX.  It's not quite as easy as
>> all that, because there aren't entries for each category separately,
>> so you'll have to parse the structure out of all of the paths in the
>> INDEX.
>
>Well, the idea I had in mind was that, if all of the individual ports'
>README.html files already are in place, then it should be trivial to
>just "ls" or "find" them under each category to fill in the category's
>README.html.  No need to reference the INDEX or anything else.  Or???
>
>The workaround method I've been running out of cron for the last month
>or so is:
>
>1) Create a "sentinel" file under /tmp to use as a timestamp, just
>before running "cvs update" on ports (I update my ports tree from a
>local copy of the CVS repo maintained via csup)
>
>2) After cvs completes, look for any port directories containing
>updates (check timestamps against the sentinel file) and do a "make
>readme" for each one:
>
>find $PORTSDIR -type f ! -path "*/CVS/*" -newercm $SENTINEL -depth 3 |
>                xargs dirname |
>                sort -u | xargs -I@ /bin/sh -c "cd @ && make readme"
>
>3) Last, but not least, build the category README.html for any
>categories with ports containing newly updated README.html files.
>
>I have noticed while doing this that, as you mentioned, the category
>READMEs take considerably longer than the individual ports'.
>
>I don't even bother to rebuild the top-level file, since it's basically
>unchanging anyway.
>
>> >> > I think the way to speed this up is to have the script generate
>> >> > the category files too. There's no point in bringing in the
>> >> > top-level README since that's already fast.
>>
>> > So what's making the category READMEs so slow then?
>>
>> The big problem with performance in all this INDEX and README.html
>> building is that it takes quite a long time relatively to run make(1)
>> within any port or category directory.  make(1) has to read in a lot
>> of other files and stat(2) many more[*] -- all of which involves a
>> lot of random-access disk IO, and that's always going to take quite a
>> lot of time.  Now, doing 'make readme' in a category directory
>> doesn't just run make in that directory, but also in every port in
>> that category. Popular categories can contain many hundreds of ports.
>
>I'm a little rusty on the actual mechanics of make, but shouldn't it be
>possible to run a single, over-arching make on each category that
>wouldn't need to spawn a bunch of sub-makes?
>
>> Maybe I should add README.html generation to my FreeBSD::Portindex
>> stuff.  Should be pretty simple -- all the necessary bits are readily
>> available and it is just a matter of formatting it as HTML and
>> printing it out.
>
>"Maybe"?  Whaddya mean, "maybe"?  :-)  Sounds like it would definitely
>be worth doing!
>
>> 	Cheers,
>>
>> 	Matthew
>>
>> [*] Running 'make -dA' with maximum debug output is quite
>> enlightening, as is running make under truss(1)
>
>Enlightening, perhaps.  Sometimes overwhelming, is more like it.  :-)
>

Not to fancy, but I used this when I was updating the readmes to not break.

#!/bin/sh
cd /usr/ports
for i in `make -V SUBDIR |sed s/local//g`; do for p in `make -C $i -V SUBDIR`; do echo $i/$p && sudo  make -C "$i/$p" readme ; done; done >> ~/readmes.log

-jgh

-- 
Jason Helfman
System Administrator
experts-exchange.com
http://www.experts-exchange.com/M_4830110.html
E4AD 7CF1 1396 27F6 79DD  4342 5E92 AD66 8C8C FBA5