what next for the pkg_install rewrite

[Ivan Voras]

On 20/08/2010, Garrett Cooper <gcooper at freebsd.org> wrote:

> 1. SQLite was killed before because of complexity and because it was
> needs another package in base that isn't BSD licensed. That's why

And... both ideas are completely wrong. SQLite can be imported as a
single C file + header, which you must agree is practically the
optimum, and its license is "public domain" which is, if anything,
"freer" than BSDL and eminently compatible with it.

> everyone in the know has been pushing for BDB 1.8x (in base). People

BDB in base offer exactly one (1) single "feature", if you can call it
that: storing and retrieving key-value binary blobs. It has no
practical concurrency support, it's locking is laughable and upgrading
data stored as binary blobs is even more nightmarish than maintaining
the current plist format (and it will lead to similar uglyness soon -
rather than upgrading the data piece called "x", I'm sure developers
will introduce new keys called "x_extdata", "x_moredata" etc etc).

SQLite on the other hand solves all these in the following way:

1) stores proper records, not blobs
2) has proper locking & concurrency support, ACID
3) the database schema can be modified on the fly for upgrading -
fields can be added to existing table while preserving data.
4) is endian-agnostic, 32/64-bit agnostic and portable (I mean the
database file) to an extremely large number of platforms. It is
already used as a system database in OS X, iPhone and Android.

Note that I'm promoting SQLite to replace the /var/db/pkg/* tree of
directories and files with ad-hoc structure - all data in there should
be in one SQLite database, which is stored in a single file.

> suggested that to me back when I was trying to get off the ground in
> GSoC 2006, they did it to you before Ivan, and I'm sure they've done
> it before in the past. We need to implement things in terms of BDB
> first, but make the APIs generic enough so that it _could_ be extended
> to SQLite if people chose to do the work later on.

Not planning an API for a transactional database in the first place
will bring the whole thing down in the long term, and in any case will
certainly push things 10 more years in the future. Note that SQLite
can *not* be considered a drop-in replacement for BDB (any version of
BDB) because in that case you will gain far less than is optimal.

If you haven't used SQLite yet, please try it, from C, and then see if
you can reevaluate your comment on its complexity. If you don't like
SQL, this is also fine. You are not out of the game, there are many
other parts to work on.

As for backward compatibility: basically it can be done in two ways:
1) build a one-time converter that will read the present plist
database and output a new database or 2) build a compatibility layer
which would support both the old and the new format at the same time,
so people upgrading will continue to use their old data. I consider
the latter wasteful in terms of developer resources and because
bit-rot will eventually make support for the old format decrepit.

> 2. XML is a bad idea. Great in theory, wonderful in my browser, but a
> bloated plaintext file with a lot of complexity. I would prefer a
> database solution that could be dumped to a simple text file format if
> needed.

XML, at least in my proposal, would not be used for the system package
database, but would replace (either in part or all with a single file)
today's "+" files in the packages, together with other purposes where
some metadata transfer is required.

That said, I would also be happy with other self-described formats
like JSON. XML is the front runner because it's more standard
(industry standard, whether someone likes this fact or not!) and there
is already a parser in base.

> Again (and I can't overemphasize this): no matter what we do, say,
> etc, unless we have buy-in from portmgr beforehand on anything here,
> the work will never see a ports/src CVS/svn repo, etc. This includes
> work done for past GSoCs, and current GSoCs.

With all respect to portmgr, I expect it to keep a cool head and
acknowledge the following:

1) Decisions must be made on objective terms which include
consideration for clean / elegant implementation, long-term
maintainability and standards. If there is to be a rewrite, it must be
built to stand the test of next 10 years of usage, and not start
compromising even before it is started.

2) Except in extreme cases, portmgr should decide based on
functionality and not have that much say in the specifics of the
implementation. Basically, the portmgr should in the first place
approve the feature spec, and the src people should worry about the
details of what can and cannot be done. This is not a src vs portmgr
war, this is separation of duty. (of course there are overlaps in
people's memberships)