Rearchitecting pkg_install (was Re: Using shell commands versus
youshi10 at u.washington.edu
Sun Jun 17 22:20:41 UTC 2007
Tim Kientzle wrote:
>>> I've also seen a 3x speedup by using my reimplementation of pkg_add
>>> using my package management library, libpkg (http://libpkg.berlios.de).
>>> It is not production ready yet as if it fails partway through an
>>> installation it won't clean up and installed files.
> I *think* a good way to do this is to:
> 1) Write each file to a temporary name (as Peter Jeremy
> suggested earlier in this thread)
> 2) Write a +INPROGRESS file to the package dir with
> the temporary name of each file as it is written.
> 3) Recursively install dependencies.
> 4) Rename the files to their final names.
> This sequence allows you to:
> * Do conflict checking on the fly. If a single file
> conflicts at any point, you can detect that, ask
> the user for a resolution at that point and drop
> the single file or back out the entire package.
> * Recover crashed installations. Just scan for all
> +INPROGRESS files and remove every file mentioned.
> There are a few additional steps you can take to provide
> even better failure recovery, but this covers the big
>>>> I haven't looked closely at pkg_delete, but I doubt there's
>>>> much that can be done to speed it up...
>>> I ran a quick test ... difference was too small to be noticeable ...
> Yeah, that's not surprising.
>> 2. I can't go and graft your libs, or do something similar with the
>> current source because I don't want to break production code
>> (pkg_install) of this importance and muck up a lot of user's systems
> If you can get enough people involved in testing, you may
> be able to pull off a full rearchitecture. Requires some
> careful political groundwork, though. ;-) Be friendly, talk about
> your work in different mailing lists, make snapshot versions
> of your code available, try to build a reputation as someone
> who responds quickly to reported problems. I managed to get
> support for replacing 'tar' this way, which is arguably just
> as critical as pkg_install.
> Tim Kientzle
Well, this certainly changes the problem description from what I
thought it was quite a bit.
I need to seriously brainstorm what needs to be done for a complete
rewrite then, and talk to Kirill, and some of the other main players in
the ports/pkg maintainers groups.
For now I'm going to run quite a few iterations with a gprof'ed copy
of my modified binaries and see if there's another point where things
are slow, other than some of the code in deps.c that Stephen's
submitted: <http://www.freebsd.org/cgi/query-pr.cgi?pr=112630>, and
speed it up as much as possible.
More information about the freebsd-hackers