Time to abandon recursive pulling of dependencies?
Stephen Montgomery-Smith
stephen at math.missouri.edu
Wed May 16 14:29:48 UTC 2007
Stephen Montgomery-Smith wrote:
> Stephen Montgomery-Smith wrote:
>> [LoN]Kamikaze wrote:
>>> Stephen Montgomery-Smith wrote:
>>>> Basically I think we are stuck on making "make package-depends" go any
>>>> faster.
>>>>
>>>> However I do think that the modifications I made to pkg_create go a
>>>> very
>>>> significant way to solving the problem of registration taking so
>>>> very long.
>>>>
>>>> Stephen
>>>>
>>>
>>> You are right about invoking make being the problem. However, I was
>>> able to
>>> increase the performance of package-depends by 20 to 30 percent, by
>>> avoiding
>>> recursive make calls and moving the recursion into the stack of the
>>> shell
>>> invocation.
>>>
>>> With my changes there's less than 10% overhead over the sum of make
>>> calls. I.e.
>>> for 56 seconds of make calls (x11/xorg on my system) there are only
>>> 4 more
>>> seconds of processing, instead of 15 to 20 seconds.
>>>
>>> The original make package-depends does some things I don't
>>> understand. Once
>>> I've figured them out I will post my little patch here.
>>>
>>>
>>
>>
>> Someone pointed out that what I was proposing in +DEPENDENCIES is
>> already to be found in +CONTENTS. So here is a proof of concept patch
>> to /usr/ports/Mk/bsd.port.mk (proof of concept because no error
>> checking, and things like that).
>>
>> For me it makes registration about 3 times faster.
>
> And also, the only reason it goes slow is because it has to do
> (cd $$dir; make -V PKGNAME)
> for every dir in _LIB_RUN_DEPENDS. But if instead we kept a file in
> /var/db/pkg called something like +PACKAGE_NAMES, where as each port is
> created we add in a one line entry with this mapping of origin to
> package name, then registration would take less than a second. It also
> has the advantage that the created +REQUIRED_BY really would be an
> accurate reflection of the true state of affairs rather than what it
> should be (meaning that if you cvsup ports and then build on top of old
> dependencies, +REQUIRED_BY would still be accurate).
>
> Stephen
Another option is to create this +PACKAGE_NAMES file on the fly,
something like as follows:
@rm -f /tmp/tempfile; \
for pkgname in /var/db/pkg/*; do \
echo `basename $$pkgname``grep "@comment ORIGIN:" $$pkgname/+CONTENTS
| sed "s/@comment ORIGIN//"` >> /tmp/tempfile; \
done; \
This process really is rather quick, maybe a few seconds, and certainly
way quicker than the many invocations of make.
More information about the freebsd-ports
mailing list