Time to abandon recursive pulling of dependencies?
Garrett Cooper
youshi10 at u.washington.edu
Sat May 19 23:43:26 UTC 2007
Stephen Montgomery-Smith wrote:
> Alexander Leidinger wrote:
>> Quoting Robert Noland <rnoland at 2hip.net> (Wed, 16 May 2007 18:14:01
>> -0400):
>>
>>> On Wed, 2007-05-16 at 16:01 -0500, Stephen Montgomery-Smith wrote:
>>>> Ok chaps, I think I have it.
>>>>
>>>> This involves no recursive calls of make. Furthermore the
>>>> dependencies it creates are the real dependencies on your system,
>>>> not what ports thinks it should be, because it gets all the
>>>> information from /var/db/pkg. On my system it takes a second or two
>>>> to register a port
>>>> - it takes about the same amount of time whether it has few
>>>> dependencies or many, except when there are zero dependencies, when
>>>> it takes no
>>>> time at all.
>>>>
>>>> If I get some positive feedback on this one, I'll submit a PR. You
>>>> can try it out by typing "make actual-package-depends" as opposed to
>>>> "make package-depends."
>>> Ok, I'm really trying to make sense of these numbers...
>>>
>>> For gnome2-2.18.1_1 on my -current system...
>>>
>>> rnoland-ibm% make package-depends|wc -l
>>> 362
>>> rnoland-ibm% make actual-package-depends|wc -l
>>> 294
>>>
>>> Registered the old way:
>>> rnoland-ibm% grep @pkgdep /var/db/pkg/gnome2-2.18.1_1/+CONTENTS|wc -l
>>> 176
>>>
>>> and the new way:
>>> rnoland-ibm% grep @pkgdep /var/db/pkg/gnome2-2.18.1_1/+CONTENTS|wc -l
>>> 294
>>>
>>> I put together an awk script yesterday which was very close to the one
>>> Alexander posted and got 295... and the diff of the output of my awk and
>>> actual-package-depends is much more than a single line, which I also
>>> can't yet explain.
>>
>> I had a look at it. It is missing those dependencies which are not
>> installed. Depending on when this target is used, this is ok or not.
>>
>> Some benchmark:
>> ---snip---
>> [original]
>> make package-depends 80.18s user 53.92s system 73% cpu 3:01.53 total
>> make package-depends 80.70s user 52.94s system 87% cpu 2:32.25 total
>> make package-depends > /tmp/list1 79.58s user 54.28s system 87% cpu
>> 2:32.59 total
>>
>> [no AWK]
>> make actual-package-depends 4.76s user 7.46s system 57% cpu 21.082 total
>> make actual-package-depends 4.87s user 7.34s system 67% cpu 18.019 total
>> make actual-package-depends > /tmp/list2 4.58s user 7.60s system 87%
>> cpu 13.942 total
>>
>> [AWK]
>> make actual-package-depends 3.15s user 7.45s system 85% cpu 12.420 total
>> make actual-package-depends 3.13s user 7.49s system 84% cpu 12.524 total
>> make actual-package-depends 3.11s user 7.49s system 88% cpu 11.995 total
>> make actual-package-depends >| /tmp/list3 3.16s user 7.42s system 89%
>> cpu 11.83 0 total
>>
>> [AWK + partly missing (only direct dependencies)]
>> make actual-package-depends > /tmp/list4 4.60s user 7.63s system 90%
>> cpu 13.479 total
>> make actual-package-depends 4.68s user 7.56s system 87% cpu 13.985 total
>> make actual-package-depends 4.58s user 7.64s system 89% cpu 13.594 total
>>
>> ...sorting list1 & list2 & list3 & list4...
>>
>> % wc -l /tmp/list*s
>> 320 /tmp/list1s
>> 308 /tmp/list2s
>> 308 /tmp/list3s
>> 310 /tmp/list4s
>> 1246 total
>> ---snip---
>>
>> Note, there was trackerd running in the background while testing...
>>
>> For the difference between the redirected output case: I think the
>> gnome terminal needs a lot of time to print all the lines. But still,
>> the awk version takes around 3/4 of the time (interesting is the user
>> time, not the total time). Stephen's version can be speed up some
>> fractions by inserting a break into the first while-loop.
>>
>> Attached is my awk-version and the awk version which also includes the
>> direct dependencies.
>>
>> Bye,
>> Alexander.
>>
>>
>
> I think I like Alexander's awk version better than mine. Alexander -
> can I ask you to submit the PR using send-pr?
>
> Stephen
My lord.. now I see what everyone means in terms of taking a long time
to update the ports / package databases. If you use portsnap, it doesn't
take a long time. However, if you use csup/cvsup, it appears to take a
long time running make and ruby (primarily disk I/O), to update all of
the databases. It took approximately 3-5 minutes to complete on a P4 3.0
GHz, so I'd hate to see how long it takes to finish on a slower machine
=\...
Maybe this is where Stephen's revisions to pkg_add and bsd.ports.mk
would come in handy.
-Garrett
More information about the freebsd-ports
mailing list