New FreeBSD package system (a.k.a. Daemon Package System (dps))

[Stanislav Sedov]

On Sat, 12 May 2007 11:31:59 -0700
Bert JW Regeer <xistence at 0x58.com> mentioned:
>
> SQLite is compiled inside a program, and as such does not require any
> resources other than one file handle and some CPU time when querying.
> The file is stored on disk, and requires no separate process to be
> running to query. Maybe I misunderstood what you were trying to say.
> SQLite will require less resources than flat text files, since SQLite
> is a one time open then process, instead of what is currently
> happening, having to open and close hundreds of files depending on
> how many ports are installed. With this regard, SQLite is like BDB.
> Where SQLite uses standards compliant SQL statements to get data.
>

It isn't a feature of SQLite per-se, it's a feature of using a single db
file instead of a bunch of files/dirs. But again, a single file is
much simpler to occasionally drop/lose/etc and it's very hard to
restore if something goes wrong. Having a backup old-style database
with a single db file, that hashes information from it will bring bot a
fast operation (since all lookups will be done in database, not flat
files) as well as an ability to easily restore database or fix it by
rebuilding from original files. Writing metadata to both location will
not require much resources obviously.

>
> I am able to understand many of the gripes with using a databases,
> and have to import yet another code base into the FreeBSD base,
> however as one of the young ones, and knowing sed/awk/grep and SQL, I
> prefer SQL over having to process hundreds of text files using text
> processing tools. It saddens me each time I run one of the pkg_*
> tools that needs to parse the flat file structure since it takes so
> long. I have friends running Ubuntu and their apt-get returns results
> much faster.
>
> In a world where hard drives are becoming more reliable, and are
> automatically relocating sectors that go bad, do we really have to
> worry about database corruption as much? I feel that many of the
> fears that are being put forward will do harm to a text based
> "storage" system as well. If one block drops out, it can cause tools
> to not be able to parse the files. Create a backup copy of the
> database after each successful transaction? There are ways to battle
> data corruption.
>
> Using BDB is not an real option either. I can not even count the
> amount of times that the BDB database that portupgrade created has
> become corrupt because I accidently ran two portupgrades at the same
> time, or even remembered that I did not want to upgrade something and
> hit Ctrl+C. The experience I got from running SVN with BDB as the
> back-end database to store my data, I say no thanks. In that case I
> would much rather stick with the flat text files than go with a
> database.
>

Every thing can be implemented both in good and bad way, it's not a
problem of bdb at all. Furthermore, we probably doesn't need a very
complex database structure, just a hashed info from current db/pkg
database. For concurrent access we can even use a simple file-based
locking, since concurrent writing to pkg database doesn't seems useful
operation to me. We can even use cdb to hash this information, and
rebuild cdb file each time it modified (not by rebuilding an entire
database, but by using information from previous cdb copy and adding
new info to it). This solution ll be reasonable fast (in fact it will
not differ much from SQLite solution at all), and will have transaction
safety.

Just IMO, anyway. Of course, a transaction safe database will be a
better solution anyway, but please - not SQL, it's complex and we don't
need it for a such simple task. Probably, we can add transactions to
our bdb 1.85, it seems not very hard to me.

--
Stanislav Sedov
ST4096-RIPE
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20070513/acceb15a/attachment.pgp