Adding .db support to pkg_tools

[Anders Nore]

On Fri, 09 May 2008 14:43:08 +0200, Joerg Sonnenberger  
<joerg at britannica.bec.de> wrote:

> On Fri, May 09, 2008 at 01:52:46PM +0200, Anders Nore wrote:
>> I'm working on adding .db support to the pkg_tools( i.e. pkg_add,  
>> pkg_info,
>> etc. ) as part of SoC 2008. The database api used is BerkeleyDB that  
>> comes
>> with the base system (/usr/src/include/db.h). BerkeleyDB is not you're
>> typical relational db, and can only save key/value pairs. The way I'm
>> thinking of storing information to the .db is to name the keys as the
>> directory names in /var/db/pkg. And save the +* files in the  
>> directories to
>> the value element in the db, separated with a special character or  
>> similar.
>
> As one of the persons hacking on pkg_install in pkgsrc/NetBSD, I would
> *strongly* advisy you against storing the files only in a bdb file.
> The change of major and complete corruption with bdb185 is high,
> consider pulling the plug in the middle of a long update.

Yes that would probably be bad for the database, but I'm sure one can  
manage to get around this problem by copying it before changing the db and  
delete the copy if it doesn't fail. At the next time executed it will  
check for a copy, use that and assume that the last run was unsuccessful.

Note that this is not a replacement for the existing tools, although it  
might be in the future. It's a standalone replica of the old tools with  
bdb support and other smaller improvements. And for compatibility reasons  
the tools will use a hybrid method (saving to both .db and flat db), but  
reading and querying will only be done on the .db file.

> Secondly, I would also advisy against just storing all meta data in a
> single key/value pair. For example, +CONTENTS can be extremely large.
> Check texmf for a good example.

When it comes to storing large values in a key/value pair, I think that's  
what bdb was designed for, handling large amounts of data (in the orders  
of gigabytes even in key's) fast.

-Anders