Time to abandon recursive pulling of dependencies?

Stephen Montgomery-Smith stephen at math.missouri.edu
Wed May 16 14:29:48 UTC 2007


Stephen Montgomery-Smith wrote:
> Stephen Montgomery-Smith wrote:
>> [LoN]Kamikaze wrote:
>>> Stephen Montgomery-Smith wrote:
>>>> Basically I think we are stuck on making "make package-depends" go any
>>>> faster.
>>>>
>>>> However I do think that the modifications I made to pkg_create go a 
>>>> very
>>>> significant way to solving the problem of registration taking so 
>>>> very long.
>>>>
>>>> Stephen
>>>>
>>>
>>> You are right about invoking make being the problem. However, I was 
>>> able to
>>> increase the performance of package-depends by 20 to 30 percent, by 
>>> avoiding
>>> recursive make calls and moving the recursion into the stack of the 
>>> shell
>>> invocation.
>>>
>>> With my changes there's less than 10% overhead over the sum of make 
>>> calls. I.e.
>>>  for 56 seconds of make calls (x11/xorg on my system) there are only 
>>> 4 more
>>> seconds of processing, instead of 15 to 20 seconds.
>>>
>>> The original make package-depends does some things I don't 
>>> understand. Once
>>> I've figured them out I will post my little patch here.
>>>
>>>
>>
>>
>> Someone pointed out that what I was proposing in +DEPENDENCIES is 
>> already to be found in +CONTENTS.  So here is a proof of concept patch 
>> to /usr/ports/Mk/bsd.port.mk (proof of concept because no error 
>> checking, and things like that).
>>
>> For me it makes registration about 3 times faster.
> 
> And also, the only reason it goes slow is because it has to do
> (cd $$dir; make -V PKGNAME)
> for every dir in _LIB_RUN_DEPENDS.  But if instead we kept a file in 
> /var/db/pkg called something like +PACKAGE_NAMES, where as each port is 
> created we add in a one line entry with this mapping of origin to 
> package name, then registration would take less than a second.  It also 
> has the advantage that the created +REQUIRED_BY really would be an 
> accurate reflection of the true state of affairs rather than what it 
> should be (meaning that if you cvsup ports and then build on top of old 
> dependencies, +REQUIRED_BY would still be accurate).
> 
> Stephen

Another option is to create this +PACKAGE_NAMES file on the fly, 
something like as follows:

	@rm -f /tmp/tempfile; \
	for pkgname in /var/db/pkg/*; do \
		echo `basename $$pkgname``grep "@comment ORIGIN:" $$pkgname/+CONTENTS 
| sed "s/@comment ORIGIN//"` >> /tmp/tempfile; \
	done; \

This process really is rather quick, maybe a few seconds, and certainly 
way quicker than the many invocations of make.



More information about the freebsd-ports mailing list