A new and better way to do "make readmes"?

[Conrad J. Sabatier]

[ Sorry to be so late in following up on this; lost track for a while ]

On Sat, 28 Jan 2012 16:53:17 +0000
Matthew Seaman <m.seaman at infracaninophile.co.uk> wrote:

> On 28/01/2012 16:28, Conrad J. Sabatier wrote:
> > <rubbing eyes in disbelief>  Am I understanding you correctly?  Are
> > you saying you built 20,000+ port READMEs in only 9 seconds?!  How
> > is that possible?  Or do you mean 9 seconds for each one?
> 
> 9 seconds sounds quite reasonable for generating 23000 or so files.

It sounds incredible to me!  :-)

> >> > Selective updating isn't going to help because 99.9% of the time
> >> > is spent in the categories and it only takes a single port
> >> > update to make a category file obsolete.
> 
> > This is the part I find troubling.  It would seem that it should be
> > more work to create an individual port README, with its plucking the
> > appropriate line out of the INDEX-* file and then parsing it into
> > its respective pieces and filling in a template, than to simply
> > string together a list of references to a bunch of already built
> > port READMEs into a category README.
> > 
> > What am I not getting here?
> 
> No -- you're quite right.  You could generate the category README.html
> files entirely from the data in the INDEX.  It's not quite as easy as
> all that, because there aren't entries for each category separately,
> so you'll have to parse the structure out of all of the paths in the
> INDEX.

Well, the idea I had in mind was that, if all of the individual ports'
README.html files already are in place, then it should be trivial to
just "ls" or "find" them under each category to fill in the category's
README.html.  No need to reference the INDEX or anything else.  Or???

The workaround method I've been running out of cron for the last month
or so is:

1) Create a "sentinel" file under /tmp to use as a timestamp, just
before running "cvs update" on ports (I update my ports tree from a
local copy of the CVS repo maintained via csup)

2) After cvs completes, look for any port directories containing
updates (check timestamps against the sentinel file) and do a "make
readme" for each one:

find $PORTSDIR -type f ! -path "*/CVS/*" -newercm $SENTINEL -depth 3 |
                xargs dirname |
                sort -u | xargs -I@ /bin/sh -c "cd @ && make readme"

3) Last, but not least, build the category README.html for any
categories with ports containing newly updated README.html files.

I have noticed while doing this that, as you mentioned, the category
READMEs take considerably longer than the individual ports'.

I don't even bother to rebuild the top-level file, since it's basically
unchanging anyway.

> >> > I think the way to speed this up is to have the script generate
> >> > the category files too. There's no point in bringing in the
> >> > top-level README since that's already fast.
> 
> > So what's making the category READMEs so slow then?
> 
> The big problem with performance in all this INDEX and README.html
> building is that it takes quite a long time relatively to run make(1)
> within any port or category directory.  make(1) has to read in a lot
> of other files and stat(2) many more[*] -- all of which involves a
> lot of random-access disk IO, and that's always going to take quite a
> lot of time.  Now, doing 'make readme' in a category directory
> doesn't just run make in that directory, but also in every port in
> that category. Popular categories can contain many hundreds of ports.

I'm a little rusty on the actual mechanics of make, but shouldn't it be
possible to run a single, over-arching make on each category that
wouldn't need to spawn a bunch of sub-makes?

> Maybe I should add README.html generation to my FreeBSD::Portindex
> stuff.  Should be pretty simple -- all the necessary bits are readily
> available and it is just a matter of formatting it as HTML and
> printing it out.

"Maybe"?  Whaddya mean, "maybe"?  :-)  Sounds like it would definitely
be worth doing!

> 	Cheers,
> 
> 	Matthew
> 
> [*] Running 'make -dA' with maximum debug output is quite
> enlightening, as is running make under truss(1)

Enlightening, perhaps.  Sometimes overwhelming, is more like it.  :-)

-- 
Conrad J. Sabatier
conrads at cox.net