Yet another INDEX builder.

[Michel Talon]

On Sat, Sep 30, 2006 at 06:14:26AM +1000, Peter Jeremy wrote:
> On Fri, 2006-Sep-29 19:21:22 +0200, Michel Talon wrote:
> >timings (*) one gets. On my machine, a P4 3Ghz, oldish, it takes 23 minutes to
> >complete building the INDEX, but the python script doesn't run more than 1
> >minute. Everything else is IO, i think. So one cannot expect any performance
> >improvement by writing such a program in C.
> 
> IMHO, the advantage of using C is that it would remove the need for
> yet another interpreter to be installed to manage my ports.  Or, to
> put is another way, by writing it in python, you have made your script
> less attractive to me than it would be if it was in C/sh/awk.
> 

I completely agree with you on that. I see python here as a prototyping tool 
to be able to write quickly something that works, so that people may easily
understand it, and hack it if they want. Assuming, and we are very
far from that, this tool seems useful, then it may be time to think about
rewriting it in C. Writing directly in C, without knowing if it has any real
interest would be a loss of time. Writing in python has many advantages:
- it is very easy to read
- it can be readily translated in C assuming availability of libraries helping
to simplify the things that python does very simply. I am here thinking of the
boost libraries, but you will object this is still another port.
- there are no high brain constructions such as in ruby and frequently in
perl, so the way to translate the logic to C is completely direct.
- python and perl are performant, ruby is dog slow.
- if you have some big package installed such as Gnome or KDE
you have python installed as a dependency. You have perl installed as soon as
you install the most innocent ports, even xorg does that if i remember well.
- finally python comes with a complete set of tools, so i don't have used any
supplementary port besides python. No database connector, nothing. This is
knowing the problems which afflicted portupgrade when you upgrade ruby
or the database connector. Python is a stable language, scripts that work will
not break after an upgrade. Coredumps you don't see.

> Several years ago, the Project went to a lot of effort to re-writes
> bits of the base system in C/sh/awk to allow perl to be removed.
> Ports-based maintenance tools are (of course) free to use any language
> but the proliferation of different tools using different languages
> means that I wind up with a large number of languages and interpreters
> installed just to easily manage my system:
> - CVSup needs Modula-2

Not really, only to build cvsup, not to run it. With a precompiled package 
you dont need Modula-2. But i am precisely advocating the idea that one should
have a reliable and easy way to upgrade using precompiled packages, and ports
only as a fallback, or in case one wants to tweak something. As soon as you
are using ports, speed or efficiency doesn't matter any more, since you will
obviously spend an enormous of time compiling, making the time spent in the
tool negligible. Hence a tool like portmaster will be amply sufficient to
fill this need. It is when you try to upgrade with precompiled packages that
you discover that portupgrade is intolerably slow.

> - portupgrade needs Ruby
> - your new index builder needs Python
> - various other ports management tools use perl

Almost any use of the ports system uses perl. There are several perl
invocations in bsd.port.mk. For example in make describe. The idea of my
script is to remove all tools forked by make index, innumerable copies of
perl, sed, and whatever else forked at any instant, by just one python 
program doing the job, and just the minimal number of forks, one fork of make
-V in each port directory. Obviously this alone fork  consumes 
almost all the run time of the program.

> 
> Please don't take the above as a personal attack - I'm merely pointing
> out that there are other reasons apart from performance for using C.

Once more i agree with you. 

> 
> -- 
> Peter Jeremy



-- 

Michel TALON