svn commit: r250991 - in head: contrib/jemalloc/include/jemalloc include lib/libc/stdlib/jemalloc

[Peter Wemm]

On Mon, Jun 3, 2013 at 7:50 AM, Marcel Moolenaar <marcel at xcllnt.net> wrote:
>
> On Jun 3, 2013, at 6:25 AM, Dimitry Andric <dim at FreeBSD.org> wrote:
>
>> On 2013-05-25 20:59, Marcel Moolenaar wrote:
>>> Author: marcel
>>> Date: Sat May 25 18:59:11 2013
>>> New Revision: 250991
>>> URL: http://svnweb.freebsd.org/changeset/base/250991
>>>
>>> Log:
>>>   Make the malloc(3) family of functions weak and make their non-weak
>>>   implementations visible for use by applications. The functions $F that
>>>   are now weak symbols are:
>>>      allocm, calloc, dallocm, free, malloc, malloc_usable_size,
>>>      nallocm, posix_memalign, rallocm, realloc, sallocm
>>>
>>>   The non-weak implementations of $F are exported as __$F.
>>
>> Hi Marcel,
>>
>> This commit seems to lead to various problems in ports, and possibly
>> other software.  For example, with Firefox I now always get a few
>> messages like:
>>
>>  firefox in free(): warning: malloc() has never been called
>>
>> Another example is the devel/talloc port, which now dies with:
>
> It looks like the python build is broken. The net effect is that _ctypes.so
> has a strong definition of malloc, free, et al. This is not intentional,
> but a side-effect of the what seems to be a bug in fficonfig.py.in (see
> patch below). To elaborate:
>
> _ctypes.so incorporates the libffi functionality for what I presume is
> the basis for Python bindings. libffi includes dlmalloc.c, an open source
> allocator. dlmalloc.c is incuded by closures.c and closures.c defines
> USE_DL_PREFIX. On top of that closures.c makes all allocator functions
> static. This, by design there's no problem. In short: dlmalloc.c never
> gets compiler separately/independently.
>
> The python build however compiles dlmalloc.c separately/independently.
> As such, dlmalloc.c now defines and exports malloc et al and it also
> get linked into _ctypes.so.
>
> Once _ctypes.so gets dynamically loaded, things start to break.
>
> Unfortunately, this means python needs to be recompiled from ports with
> the following fix:
>
> Index: files/patch-Modules-_ctypes-libffi-fficonfig.py.in
> ===================================================================
> --- files/patch-Modules-_ctypes-libffi-fficonfig.py.in  (revision 0)
> +++ files/patch-Modules-_ctypes-libffi-fficonfig.py.in  (working copy)
> @@ -0,0 +1,10 @@
> +--- Modules/_ctypes/libffi/fficonfig.py.in.orig        2013-06-03 07:16:44.000000000 -0700
> ++++ Modules/_ctypes/libffi/fficonfig.py.in     2013-06-03 07:17:03.000000000 -0700
> +@@ -1,7 +1,6 @@
> + ffi_sources = """
> + src/prep_cif.c
> + src/closures.c
> +-src/dlmalloc.c
> + """.split()
> +
> + ffi_platforms = {
>
> This has been tested with python-2.7.5. I can't say anything about
> other versions.
>
> Do people concur that this is the right fix?

Forgive me for asking dumb questions, but:

* Does this fix the firefox, chromium etc problems?  (were they linked
to python?)

* Even if the python code is wrong, is this one of those things that's
going to keep biting us going forward?  Interactions between
dlopen/dlsym/rtld/library dependencies/embedded malloc
replacements/etc has been accumulated a large number of casualties
over the years.

-- 
Peter Wemm - peter at wemm.org; peter at FreeBSD.org; peter at yahoo-inc.com; KI6FJV