Rearchitecting pkg_install (was Re: Using shell commands versus C equivalents)

[Tim Kientzle]

>> I've also seen a 3x speedup by using my reimplementation of pkg_add
>> using my package management library, libpkg (http://libpkg.berlios.de).
>> It is not production ready yet as if it fails partway through an
>> installation it won't clean up and installed files.

I *think* a good way to do this is to:
    1) Write each file to a temporary name (as Peter Jeremy
       suggested earlier in this thread)
    2) Write a +INPROGRESS file to the package dir with
       the temporary name of each file as it is written.
    3) Recursively install dependencies.
    4) Rename the files to their final names.

This sequence allows you to:
  * Do conflict checking on the fly.  If a single file
    conflicts at any point, you can detect that, ask
    the user for a resolution at that point and drop
    the single file or back out the entire package.
  * Recover crashed installations.  Just scan for all
    +INPROGRESS files and remove every file mentioned.

There are a few additional steps you can take to provide
even better failure recovery, but this covers the big
concerns.

>>> I haven't looked closely at pkg_delete, but I doubt there's
>>> much that can be done to speed it up...
>>
>> I ran a quick test ... difference was too small to be noticeable ...

Yeah, that's not surprising.

> 2. I can't go and graft your libs, or do something similar with the 
> current source because I don't want to break production code 
> (pkg_install) of this importance and muck up a lot of user's systems 
> irrevocably.

If you can get enough people involved in testing, you may
be able to pull off a full rearchitecture.  Requires some
careful political groundwork, though. ;-)  Be friendly, talk about
your work in different mailing lists, make snapshot versions
of your code available, try to build a reputation as someone
who responds quickly to reported problems.  I managed to get
support for replacing 'tar' this way, which is arguably just
as critical as pkg_install.

Tim Kientzle