Proposal for alleviating disk read / write time

[Garrett Cooper]

Hello Kirill and Hackers,

    After reviewing the changes I made last weekend to pkg_install I'm 
seeing slight improvements, but not a large amount of improvement in 
overall program operation. I haven't implemented mmap(2) yet as John 
Baldwin suggested (need to do some reading about fcntl(2) tomorrow 
because I want to implement locking), so I should see a large boost 
then, but I was wondering if I should do something similar to the 
following to reduce redundant disk accesses.

1. Lock +CONTENTS and related files exclusively for writes. Allow shared 
reads among separate pkg_install (and derivative programs) processes, 
and derivative packing lists with a limited set and use a LRU type 
algorithm to move packing lists in and out of shared memory.
2. If shared memory amongst separate processes is possible, share the 
ports INDEX's entire package list as a global object, and update only if 
the INDEX's real and virtual copy get out of sync (i.e. local copy is 
replaced, modified, or touch(1)'ed).

    +CONTENTS and INDEX are read frequently (once per process set) and 
although modifying pkg_install won't benefit overall operation speed 
without implementing a different data structure setup (still aiming for 
something like red-black trees for larger package lists like INDEX has, 
and linked lists for the smaller package lists), I think that the above 
changes would reduce wait times as pkg_install and installing ports 
becomes more and more concurrent.

    Clarifications and comments are more than welcome. I really 
appreciate it.
    Also, if someone can point me to the unofficial locking guidelines 
page (I remember it coming up twice in the past, but I'm busy now and 
wondering if someone can quickly point me to the page), it'd be more 
than appreciated.

Thanks!
-Garrett