Trouble with Berkeley DB version 4.4.20?

[Mark]

> -----Original Message-----
> From: Alex Zbyslaw [mailto:xfb52 at dial.pipex.com]
> Sent: maandag 2 oktober 2006 12:50
> To: Mark
> Cc: 'FreeBSD-Questions Questions'
> Subject: Re: Trouble with Berkeley DB version 4.4.20?
>
>

Thanks for answering, again.

> > I actually already tried p5-BerkeleyDB too. Had high hopes for it. It
> > took a good 10 minutes or so to compile a new gcc, 3.2.23, first; but
> > after that, everything went well. Except that it shows the same behav-
> > ior. Compile goes fine; "make tests" too. But my app core dumps every,
> > say, 30 times it accesses BerkeleyDB.
>
> Not sure why p5-BerkeleyDB thinks it needs a specific gcc, though
> 3.2.23 is the version I have anyway (FreeBSD 5.4) so I didn't know to
> warn you :-(

For all the difference it made. :)

> I don't know what your apps do, but make sure for example, that if you
> can have multiple instances which can write to the db at the same time
> that your are locking correctly. In 1.85 compatibility usage that means
> locking a separate file with e.g. flock and *not* locking the database
> file itself which doesn't work right.

I know. In fact, I have one mini-Perl installed, with a DB_File compiled
against BerkeleyDB 1.85, for that precise reason (a Perl process that
shares the db with DRAC).

> PS I know there isn't a port yet, but you could just try compiling 4.5
> from the sleepycat sources. The port for 4.4 doesn't look like it does
> anything sophisticated: just applies the latest patches from sleepy-
> cat, sets up some configure args, and has one small patch to dist/con-
> figure which changes -avoid-version into -version-info 0:0:0 in a cou-
> ple of places.

Already did that; I got impatient. :) The changes to the Makefile are
minor, and I got 4.5.20 to compile and install just great.

> Waiting for 4.5 is an option, but I would worry that whatever change in
> 4.4 is making your app crash might also bite in 4.5. I could find noth-
> ing on google about perl+4.4 core dumping so I would harbour a suspicion
> that it is something about the way you are using BerkeleyDB which is
> causing the trouble.

Upgrading to 4.5 made no difference. I even installed an entirely new Perl
(5.8.8) for the occasion. To no avail.

I really suspect it's a locking issue. My app is a socketmap daemon, for
sendmail, that does SPF queries and the like. So, concurrency is
definitely happening when processes are forking.

The one time I was actually able to log an error for the process, it said:
"Invalid locker id" when opening the BerkeleyDB environment. That in
itself may not necessarily mean anything (could just mean the env has
become corrupted). But the occurance of the core dumps are consistent with
a locking issue (under heavy, concurrent load).

BerkeleyDB 0.30 (the Perl package) itself takes care of locking the shared
environment. Paul Marquess, the author of BerkeleyDB (the Perl package),
wrote me once:

"Apart from that you don't need to do anything -- Berkeley DB handles the
locking for you behind the scenes."

And it's certainly true for for 4.2: the 'tied' hashes can be accessed,
concurrently, between processes, even under the most stringent load.

Since an upgrade to 4.5 made no difference, I'm really thinking this is
just a matter of BerkeleyDB 0.30 (the Perl package) no longer doing the
locking properly. Or maybe there's something between BerkeleyDB 4.2 and
4.4 that has changed about the manner in which locks are done that
BerkeleyDB 0.30 is unaware of.

- Mark