Speeding up pkg_version and perhaps other port utilities

[Garrett Cooper]

Stephen Montgomery-Smith wrote:
> Stephen Montgomery-Smith wrote:
>>
>>
>> On Sun, 20 May 2007, Doug Barton wrote:
>>
>>> Alexander Leidinger wrote:
>>>> Quoting Stephen Montgomery-Smith <stephen at math.missouri.edu> (Sat, 
>>>> 19 May 2007 23:48:52 -0500):
>>>>
>>>>> On my system, the program pkg_version can double its speed simply 
>>>>> by replacing "make -V PKGNAME" by "make BEFOREPORTMK=yes -V PKGNAME"
>>>
>>> This is very cool! I've already added it to my devel version of 
>>> portmaster in several places. I call make a LOT in portmaster, so I 
>>> went through every invocation and added this wherever it worked 
>>> (i.e., didn't cause a difference in (or absence of) output.
>>>
>>
>> However just because it works in a majority of cases doesn't mean that 
>> it will work all the time.  You just don't know what important thing 
>> might be in AFTERPORTMK, which 99% of the time you never need.
>>
>> So I see two approaches:
>>
>> 1.  Add more .if's to bsd.port.mk so that there is a more efficient 
>> "make VARIABLEONLY=yes -V variable-name" that does all the variables, 
>> but doesn't calculate any of the targets - this might make things a 
>> bit faster, or it might not;
> 
> I don't think this will speed things up.  I hacked into the code for 
> make, putting conditionals so that when the "-V" flag was set, that make 
> wouldn't bother to calculate targets or dependencies or creations, but 
> only variables.
> 
> 
>> 2.  Profile bsd make and see if there are any bottlenecks.  I bet make 
>> was never designed for speed in these kinds of situations.  But this 
>> would be a long term project, albeit definitely worth doing.
> 
> It looks to me like the variables are stored as a linear list in the 
> "make" program.  Thus if you have something like 500 variables (e.g. try 
> a "make -d g1 -V XX | grep = | wc -l" in a port) it is going to take 
> quite some time to search through all the variables to find one.  This 
> is especially a problem for variable assignments using "?=" which first 
> has to search to see if the variable is already defined.  And there are 
> a lot of "?=" in bsd.port.mk, 298 of them.  I haven't done any profiling 
> yet, but I bet that this is what is taking up all the time.  Probably 
> the way to solve this is to rewrite src/usr.bin/make/var.c so that it 
> uses some more sophisticated way of storing the variables - maybe 
> Berkeley databases is the way to go.  I almost get the feeling that if 
> make were completely rewritten in perl that it would go faster!

Berkeley databases will be nice, but I'm not really going for the way 
that the ruby scripts do it. It's a really bad hack..

I'll try messing around with the ruby scripts and see what I can do 
about fixing that, compare the results, and see if I should go ahead 
with committing something with those scripts.

-Garrett