Berkeley DB 4.1

[Matthias Andree]

Am 13.09.2013 21:11, schrieb olli hauer:
> On 2013-09-12 12:57, Ivan Voras wrote:
>> Hello,
>>
>> I'd like to start a discussion on changing the default BDB port from 4.1
>> to something more recent.
>>
>> bdb version 4.1 was last released in 2002:
>> README: Sleepycat Software: Berkeley DB 4.1.25: (December 19, 2002)
>>
>> There are some ports which have an unexpected dependacy on bdb via APR
>> (apache22, subversion), which does is not wrong in itself, but is
>> somewhat unelegant (as a personal opinion of course).
>>
>> I've found this previous discussion:
>> http://lists.freebsd.org/pipermail/freebsd-ports/2013-February/081444.html
>>
>> And while the argument seems valid, it also doesn't have an estimate of
>> which / how many ports will break with a more recent bdb.
>>
>> Could an experimental port build be done with setting WITH_BDB_VER to
>> either the most recent 4.x version (WITH_BDB_VER=48), to the last
>> Sleepycat Licensed version (50) or the recent version in ports (60) to
>> see what breaks?
>>
> 
> Long time ago I had a short discussion with mandree@ about a bump to 4.8.
> 4.8 should be supported by most ports (also long time the default in many
> linux distributions)
> 
> At last one argument I haven't seen in the discussion is "data migration",
> for example amavisd-new provides maintenance and backup / recover routines
> but other ports ??
> 

Greetings,

I'll pick Olli's post as a stepping stone, more or less randomly.

My thoughts on this are (more or less random):


- sorry my efforts to axe anything older than db5 stalled on my side.

I always meant to have that -exp run done with - as a first step - the
default selection logic twisted in a way that anything that states "41+"
or similar meant "5+", but somehow no time felt right to me because I
shied away from meddling with releases.


- about application compatibility with the library (at build time), it
depends on how sophisticated interfaces you use.  Berkeley DB has a
rather rich feature set, is customizable, and so on.  Many applications
just use BDB as a key/value datastore.  You can, however, go all the way
to a concurrent and transactional database that gets replicated (I never
used the latter, and never received a feature request to enable it).
For applications using these more advanced features, there are more API
changes that require version comparison.  In bogofilter's earlier days
(bogofilter is also about a decade old), we had to add DB version checks
to support a new BDB version rather often, but the last half dozen or so
versions did not change function signatures that bogofilter used.


- about data migration, this is not quite as hard.  Newer library
versions are supposed to read database files written by older versions;
usually the on-disk database format remained unchanged.  What changed
often, though, is the "environment", that is the __db.001-like stuff
that keeps locking tables for concurrent access, and thereabouts.  The
log files for databases in transactional mode also changed format on
more than one occasion.  However, for applications that can be cleanly
shut down (which is more a logistic than a software problem), usually
the upgrade path is:


1. shut down application,

2. make sure that no pending corruptions are there, in doubt, run quick
recovery -- basically the idea is to roll the log file changes out into
the database, quite similar to having a journalling file system write
its journal data back into the actual structures.  This is the crucial
point - the log must have been fully processed, and all pending
transactions committed to the .db file, or rolled back.

3. remove the environment (this leaves log files intact)

4. upgrade

5. restart application.  BDB will then happily recreate the environment,
bump the logfile number, and start writing into the new.

The only thing is there's no way back unless there are database updates.

Again, I don't know about replicated databases.  This is surely in the
BDB docs somewhere and I did not care to read it ;-)

The downside is that upgrades require manual interaction, and the ports
systems is not rigged up to support the user in any way for such
upgrades (this is not specific to Berkeley DB)


- at the time when I first looked, there were ~17 ports that hardcoded a
particular BDB version, among them the high-profile openldap* ports.


- don't get me started about the abundant autoconf/configure.ac abuse
though.  Many build systems do a stupid quadratic search trying to match
a db header to a db library.  Too much, and globbing would speed things
up.  I meant to write an autoconf macro, but alas, no time for that either.


I think once 9.2 is out the door we should mark any db4* version
deprecated, and I am inclined to hack all db4* ports to refuse building
on 10-CURRENT, and make 5 the default, so that all practical 10-RELEASE
systems will be free of db4*.  Thoughts?