A new and better way to do "make readmes"?

[Matthew Seaman]

On 28/01/2012 16:28, Conrad J. Sabatier wrote:
> <rubbing eyes in disbelief>  Am I understanding you correctly?  Are you
> saying you built 20,000+ port READMEs in only 9 seconds?!  How is that
> possible?  Or do you mean 9 seconds for each one?

9 seconds sounds quite reasonable for generating 23000 or so files.

>> > Selective updating isn't going to help because 99.9% of the time is
>> > spent in the categories and it only takes a single port update to
>> > make a category file obsolete.

> This is the part I find troubling.  It would seem that it should be
> more work to create an individual port README, with its plucking the
> appropriate line out of the INDEX-* file and then parsing it into its
> respective pieces and filling in a template, than to simply string
> together a list of references to a bunch of already built port READMEs
> into a category README.
> 
> What am I not getting here?

No -- you're quite right.  You could generate the category README.html
files entirely from the data in the INDEX.  It's not quite as easy as
all that, because there aren't entries for each category separately, so
you'll have to parse the structure out of all of the paths in the INDEX.

>> > I think the way to speed this up is to have the script generate the
>> > category files too. There's no point in bringing in the top-level
>> > README since that's already fast.

> So what's making the category READMEs so slow then?

The big problem with performance in all this INDEX and README.html
building is that it takes quite a long time relatively to run make(1)
within any port or category directory.  make(1) has to read in a lot of
other files and stat(2) many more[*] -- all of which involves a lot of
random-access disk IO, and that's always going to take quite a lot of
time.  Now, doing 'make readme' in a category directory doesn't just run
make in that directory, but also in every port in that category.
Popular categories can contain many hundreds of ports.

Maybe I should add README.html generation to my FreeBSD::Portindex
stuff.  Should be pretty simple -- all the necessary bits are readily
available and it is just a matter of formatting it as HTML and printing
it out.

	Cheers,

	Matthew

[*] Running 'make -dA' with maximum debug output is quite enlightening,
as is running make under truss(1)

-- 
Dr Matthew J Seaman MA, D.Phil.                   7 Priory Courtyard
                                                  Flat 3
PGP: http://www.infracaninophile.co.uk/pgpkey     Ramsgate
JID: matthew at infracaninophile.co.uk               Kent, CT11 9PW

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 267 bytes
Desc: OpenPGP digital signature
Url : http://lists.freebsd.org/pipermail/freebsd-ports/attachments/20120128/95460d43/signature.pgp