ports-mngmt/portmaster fetch process

[jhell]

On 1/10/2010 3:41 PM, Doug Barton wrote:
> On Sat, 9 Jan 2010, jhell wrote:
> 
>>> Doug,
>>>
>>> Curious if I can get your comments about a possible change in
>>> portmaster to change the behavior of its fetch method.
>>>
>>> While currently its fetch method is optimal for a environment that
>>> has a speedy ultra-fast connection, it runs havoc on a dial line "56k
>>> with speed-boost ;)"
> 
> I'm sorry to hear that you're having problems, but you're correct that
> the assumption is that the user will have a line faster than 56k. :)
> 
Going from 6Mbps connection down to a dial line sucks but its only
temporary for now. But has also allowed me to see this behavior with
portmaster which I take as a win win for the project on my behalf and
for unsaid other users that may have slower connections too.

> 
>>> Would it be possible to: ?
>>> 1) Add a switch that changes its behavior to single fetches - or -
>>> 2) Change the default of multiple fetches to single.
>>>
>>> I opt for (2). as I really see no need have more then one fetch
>>> present at a time since it will wait for all others to finish anyway.
> 
> I'm not sure what you mean here. Currently the background fetching is
> done in parallel, and the only time portmaster waits is when the port
> it's actually trying to build hasn't finished downloading yet.
> 
You have explained this better than I could below.

>>> The problem I am noticing is that fetches that are over the amount of
>>> three consecutive connections just revolve until others have finished
>>> causing DNS lookups on a line that is already at full speed and
>>> resulting in slower transactions of the first three connections.
> 
> Not sure what you mean here either, the behavior you describe is not
> something that portmaster is doing.
> 
No this is not a problem of portmaster but more a of a problem with a
slow line that it exploits if you may. Because the line can be easily
saturated and limited to minimal concurrent number of connections a
large amount of fetches can easily make DNS lookups for new addresses
that have not been resolved not actually make it back down the line to
the client. So if fetch process 1, 2 & 3 resolve and start downloading
then the line is at its max download rate, further DNS lookups by fetch
process 4, 5 & 6 never resolve because they never make it back down the
line resulting in a host not found and the process just trying to find
the next address causing another lookup that can not be resolved until
some of the bandwidth has been reclaimed after a previous process finishes.

>>> If there is something that I have looked over in the script then
>>> please excuse me but I have not noticed a predetermined way to change
>>> this as it stands right now.
> 
> No, there is no option to affect how it does fetching at this time. I
> can think of two things that might help. The first would be for you to
> fetch the files in advance before portmaster starts trying to build
> things. Use whatever method you use to determine that there are packages
> that need updating (portmaster -L, pkg_version, etc.) and then do:
> portmaster -F pkg-one-1.23 && portmaster -F pkg-two-2.34
> 
This is effectively what I am doing now after I noticed what was
happening. pkg_version -vl'<' and get the update list and then ( cd
/usr/ports/*/updated_port ; make checksum-recursive ). But this is a lot
of manual intervention I would rather avoid. Of course I could just
write a simple script to parse the output of pkg_version and run
portmaster -F on that and if it completes successfully just run
portmaster -a.

> The other alternative that I can think of off hand is a new option to
> disable background fetching altogether. This would allow the ports
> infrastructure to handle the fetching when you got to that port in the
> build list. You'd have to wait for each port to fetch before you can
> build it, but you would be guaranteed not to have more than one fetch
> working at a time. I don't think that would be too hard to implement,
> but before I start working on it I want to be sure that it is something
> that would meet your needs.
> 
This would be perfect!. As I was searching through portmaster before I
was specificly looking through the option -v for verbose and had thought
to my self that it would be pretty cool if that functionality would
reside within that option "Just an idea".

> As for the other problem you described, are you saying that after you
> hit ^C to kill portmaster while there were background fetches happening
> that the fetch processes were still there after portmaster exited? If
> so, that's a fairly serious problem. What version of portmaster and
> FreeBSD are you using?
> 
Yes, background fetches were left after ^C of portmaster. approximately
the same amount of fetches as the amount of ports to be updated.

On stable/7, portmaster version 2.16

I am curious if this is possibly a result of the slow connection I have
as normally on a fast connection you really would not see this behavior
because all the fetches would finish rather quickly for small downloads.

This could be repeated through limiting a faster connections upload and
download speeds for dns, ftp & http protocols to 3.5kbps. And ^C
portmaster during a fetch of more then 3 ports.

> 
> Doug

Thanks Doug.

-- 

 jhell