Finding slowdowns in pkg_install (continuations of previous threads)

[Garrett Cooper]

Tim Kientzle wrote:
>>    -I tried ... buffering ...  the +CONTENTS file parsing function, 
>> and the
>> majority of the time it yielded good results ....
>
> One approach I prototyped sometime back was to use
> libarchive in pkg_add as follows:
>   * Open the archive
>   * Read +CONTENTS directly into memory (it's
> guaranteed to always be first in the archive)
>   * Parse all of +CONTENTS at once
>   * Continue scanning the archive, disposing
> of each file as it appears in the archive.
>
> Based on my experience with this, I would
> suggest you just read all of +CONTENTS
> directly into memory at once and parse
> the whole thing in a single shot.
> fopen(), then fstat() to get the size,
> then allocate a buffer and read the whole
> thing, then fclose().  You can then
> parse it all at once.
>
> As a bonus, your parser then becomes a nice
> little bit of reusable code that reads
> a block of memory and returns a structure describing
> the package metadata.
>
> Tim Kientzle
I'm not 100% sure because I'm not comparing apples (virtual disk on 
desktop via VMware) to apples (real disk on server), but I'm showing a 
2.5-fold speedup after adding the simple parser:

Virtual disk:
        4.42 real         1.37 user         1.47 sys

Real disk:
       10.26 real         5.36 user         0.99 sys

I'll run a battery of tests just to ensure whether or not that's the case.

Be back with results in a few more days.

-Garrett