Rearchitecting pkg_install (was Re: Using shell commands versus C equivalents)

[Garrett Cooper]

Tim Kientzle wrote:
>>> I've also seen a 3x speedup by using my reimplementation of pkg_add
>>> using my package management library, libpkg (http://libpkg.berlios.de).
>>> It is not production ready yet as if it fails partway through an
>>> installation it won't clean up and installed files.
>
> I *think* a good way to do this is to:
>    1) Write each file to a temporary name (as Peter Jeremy
>       suggested earlier in this thread)
>    2) Write a +INPROGRESS file to the package dir with
>       the temporary name of each file as it is written.
>    3) Recursively install dependencies.
>    4) Rename the files to their final names.
>
> This sequence allows you to:
>  * Do conflict checking on the fly.  If a single file
>    conflicts at any point, you can detect that, ask
>    the user for a resolution at that point and drop
>    the single file or back out the entire package.
>  * Recover crashed installations.  Just scan for all
>    +INPROGRESS files and remove every file mentioned.
>
> There are a few additional steps you can take to provide
> even better failure recovery, but this covers the big
> concerns.
>
>>>> I haven't looked closely at pkg_delete, but I doubt there's
>>>> much that can be done to speed it up...
>>>
>>> I ran a quick test ... difference was too small to be noticeable ...
>
> Yeah, that's not surprising.
>
>> 2. I can't go and graft your libs, or do something similar with the 
>> current source because I don't want to break production code 
>> (pkg_install) of this importance and muck up a lot of user's systems 
>> irrevocably.
>
> If you can get enough people involved in testing, you may
> be able to pull off a full rearchitecture.  Requires some
> careful political groundwork, though. ;-)  Be friendly, talk about
> your work in different mailing lists, make snapshot versions
> of your code available, try to build a reputation as someone
> who responds quickly to reported problems.  I managed to get
> support for replacing 'tar' this way, which is arguably just
> as critical as pkg_install.
>
> Tim Kientzle
    Well, this certainly changes the problem description from what I 
thought it was quite a bit.
    I need to seriously brainstorm what needs to be done for a complete 
rewrite then, and talk to Kirill, and some of the other main players in 
the ports/pkg maintainers groups.
    For now I'm going to run quite a few iterations with a gprof'ed copy 
of my modified binaries and see if there's another point where things 
are slow, other than some of the code in deps.c that Stephen's 
submitted: <http://www.freebsd.org/cgi/query-pr.cgi?pr=112630>, and 
speed it up as much as possible.
-Garrett