freebsd-update painfully slow - slower than source code build of world and kernel

[Daniel Bond]

Hi,

thanks for useful and relevant information. However, this is just one  
part of the process. Generating the diffs,
prepping the server, and the whole server-side setup/management of it  
is another - I am sure there are tools for this too.

Cheers,

Daniel.


On Jan 6, 2009, at 6:30 PM, Vincent Hoffman wrote:

> Daniel Bond wrote:
>> The same could be said about CVSup, one could write a caching cvsup
>> proxy-server, and then we could just get rid of all the other
>> cvsup-servers, except two (like freebsd-update soon will have). The
>> point is, for portsnap and freebsd-update to scale properly, it needs
>> to be opened up to the public, like CVSup is. People running a single
>> server at home, or maybee two, most like won't want to set up a PROXY
>> server, and they would be required to update both servers at the same
>> day for the Proxy server to actually cache something - which many may
>> not want. And there are a lot of people running a few servers, here
>> and there.
>>
>>
>>
>> Sure, a national squid-proxy could work - although, there is no
>> individual proxy setting for portsnap/freebsd-update.. It honors
>> HTTP_PROXY environment variable, which a lot of other tools also use.
>> Some tools might not work via this proxy, especially for local
>> addresses - the administrators of these servers probably don't want
>> all the ports tarballs to go via these, and people could use them for
>> nasty things. So, then we are back to manually setting/specifying the
>> proxy-server, each time one wants to run the commands - which people
>> might forget. (Is this getting complicated enough yet..?) We would
>> basically be creating a whole lot of new potential problems for the
>> users, to solve the problem in question..
>>
>>
>> I am also interested in learning how the portsnap protocol works,
>> maybe there are potential issues with it, that a second eye might
>> spot, or room for improvement? From what I gather, Colin is a very
>> cleaver guy, so it is not very likely, but still, other people could
>> learn from it.
>>
> well portsnap/freebsd-update are shell scripts so not too hard to  
> read.
> The actual transfer protocol is  piplined http and is done by
> /usr/libexec/phttpget  (in base so src code available
> /usr/src/usr.sbin/portsnap/phttpget/phttpget.c )
> also see http://www.daemonology.net/phttpget/
>
>
>> I would like to see these tools as the default recommended tools to
>> use in the future, and that is why I am so worried about this.
>> The point I am trying to make is, or actually the question is: Why is
>> freebsd-update (and portsnap) so secretive? Why can't the average Joe
>> run his own portsnap-mirror at home? What are we afraid of?
> I seem to remember once reading that Colin wanted to make it a more
> polished system before he release it, but i cant find that email  
> anymore.
>
> Vince
>>
>> I don't see any problems with this, except maybe loosing some detail
>> in Colin's nice graphs (which would be the case for proxies too).
>>
>>
>> Cheers,
>>
>>
>> Daniel.
>>
>>
>> On Jan 6, 2009, at 5:42 PM, Christopher Arnold wrote:
>>
>>>
>>>
>>> On Tue, 6 Jan 2009, Daniel Bond wrote:
>>>
>>>> reading your answer, you are obviously confusing what I am saying
>>>> about freebsd-update with the portsnap program. Also, I also wrote
>>>> in my first post
>>> No i'm not confusing them, just trying to follow two subjects at the
>>> same time. Sorry if that is confusing.
>>>
>>>> that HTTP_PROXY / Caching proxy server does not help me much. This
>>>> is because I download a lot of "initial tarball snapshots".. I  
>>>> would
>>>> rarely see "Cache hits" in my proxy log. I guess I could set
>>>> something up to fetch nightly via proxy, to keep the data in house,
>>>> for when I need it. I don't want to use a PROXY server, I feel this
>>>> is attacking the problem at the wrong end.
>>>>
>>> Ok, lets go again. Either you mirror (maybe by having a squid proxy
>>> and walk the tree) and thats going to me even worse for you. Or you
>>> use a squid proxy to keep stuff you need close to you and share  
>>> among
>>> different installations.
>>>
>>> Or you setup one or more national squid proxies and configure your
>>> machines manually just like you do with cvsup.
>>>
>>>
>>>
>>>> I agree, I am interested to hear the views of the wise ones.
>>>> Personally I'm going back to CVSup until freebsd-update and  
>>>> portsnap
>>>> mirrors are in a more distributed or usable state.
>>>>
>>> At least portsnap started to work for me earlier today. Havn't tried
>>> update yet.
>>>
>>> But yes i agree, update and portsnap infrastructure could be done
>>> better.
>>> I have some ideas and will try to write them down in a while.
>>>
>>>    /Chris
>>> _______________________________________________
>>> freebsd-stable at freebsd.org mailing list
>>> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
>>> To unsubscribe, send any mail to
>>> "freebsd-stable-unsubscribe at freebsd.org"
>>
>> _______________________________________________
>> freebsd-stable at freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
>> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org 
>> "
>
> _______________________________________________
> freebsd-stable at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org 
> "