Memory Error using Mailman on FreeBSD. How to debug?
mark at msapiro.net
Fri Feb 8 16:20:34 UTC 2008
Alex Zbyslaw wrote:
>You are also getting a stack trace from python when it exits with the
>"out of memory" error. ktrace is just showing python printing the stuff
>- it may be that the error also ends up in a log file somewhere - don't
>know where mailman logs, sorry. From that stack trace it should be
>possible to figure out which line of the python is actually causing that
>memory request. My bet is on one of the cPickle lines, but it would be
>nice to see the stack trace "raw" so to speak. Maybe that stack trace
>would help someone on the mailman list suggest something else.
Hi, I'm the Mailman guy on this thread.
>From the errors that were reported on mailman-users at python.org, there
are two different manifestations that we've seen, resulting in
different tracebacks (logged in Mailman's 'error' log).
Here's the flow.
With a message of 1.5 to 2 MB in size, the message is piped from the
MTA to Mailman's 'post' script which creates a queue entry which is a
file containing two python pickles. The first just contains the raw
message text and the second is a small bit of metadata about the
message/entry. This process which is a pipe spawned by the MTA works.
Next, Mailman's IncomingRunner picks up the queue entry and hits the
MemoryError exception in the call to cPickle to read the file and
unpickle the raw message. This is before it has done any processing of
the message, so the MemoryError is only related to unpickling the raw
text if the text is large enough.
With a message of 300 to 500 KB, the process is the same, but the
unpickling of the queuue entry into message text and metadata
succeeds. The MemoryError occurs in this case in the Python email
library as it is parsing the message text into an email message
object. This process is known to be a memory hog in that at some point
the entire raw message text and the full message object (which is at
least as big as the message text) will be resident at the same time.
I.e., it takes more than twice as much memory to parse a message as to
Note that these messages are not really large. Most Mailman
installations can handle much larger messages without problem.
>Did you already try sending a different kind of attachment that's the
>same kind of size (a bit bigger would be better). Maybe it's something
>about the attachment itself that's causing the issue?
In the first case above, it is almost certainly just the size of the
message text. I'm certain the same thing would occur with a simple
text/plain message if it had a 1.5MB body.
>As a final resort, if none of the above resolves or leads to clues, I
>would try uninstalling python2.5 and installing python2.4 *just in
>case*. I'm assuming that you only have python for mailman. (If you
>have real python users then it's trickier. You can install multiple
>versions of python but possibly not from ports. But python always
>compiled cleanly from tarball on FreeBSD for me. I can offer some help
>with that process if you really need it).
That would be worth a try. There could be a problem in the underlying
I don't know how FreeBSD packages Mailman, but the source distribution
installs a pythonlib directory in Mailman's tree with its own Python
email library which is compatible with Python back to 2.1, so if this
is the case with FreeBSD, downgrading Python wouldn't change the email
packege, but it still might help.
>I can't help thinking that 500Kb is a very small attachment and I can't
>really see why it would legitimately cause a request for so much memory
>that your settings aren't handling it.
>A quick look at the mailman web site shows that you can run qrunner from
>the command line - couldn't immediately find the man page though.
>you could somehow queue up the email with Mailman switched off, you
>could run qrunner by hand and then you'd definitely get the python
>backtrace. Maybe the mailman list, or a mailman admin here, can help
>with that, if you need it.
We see the Python traceback, so we don't need to do that.
Mark Sapiro <mark at msapiro.net> The highway is for gamblers,
San Francisco Bay Area, California better use your sense - B. Dylan
More information about the freebsd-questions