ffmpeg and mmap

[Andriy Gapon]

Martin,

apologies - my report was incomplete.  I really meant to include the following new
information at that time, but somehow I forgot about it and the
_BSD_VISIBLE/POSIX_C_SOURCE discussion that followed distracted me.

on 12/12/2010 22:40 Andriy Gapon said the following:
> 
> I've been getting some crash dumps in libswscale.so code.
> The stack trace is always like this:
> #0  0x000000083a6abf10 in ?? ()
> #1  0x000000080a717dc6 in hyscale_fast_MMX2
> #2  0x000000080a71bd64 in swScale_MMX2
> #3  0x000000080a71ebf9 in sws_scale
> ...
> 
> From disassembling I've identified that the crash happens as soon as inline
> assembly in hyscale_fast_MMX2 calls code pointed to by lumMmx2FilterCode pointer.
> 
> The following code in libswscale/utils.c, function sws_getContext() is of interest:
> 
> #if ARCH_X86 && (HAVE_MMX2 || CONFIG_RUNTIME_CPUDETECT)
> // can't downscale !!!
>         if (c->canMMX2BeUsed && (flags & SWS_FAST_BILINEAR)) {
>             c->lumMmx2FilterCodeSize = initMMX2HScaler(      dstW, c->lumXInc,
> NULL, NULL, NULL, 8);
>             c->chrMmx2FilterCodeSize = initMMX2HScaler(c->chrDstW, c->chrXInc,
> NULL, NULL, NULL, 4);
> 
> #ifdef MAP_ANONYMOUS
>             c->lumMmx2FilterCode = mmap(NULL, c->lumMmx2FilterCodeSize,
> PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
>             c->chrMmx2FilterCode = mmap(NULL, c->chrMmx2FilterCodeSize,
> PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);

The lines above look differently in the ffmpeg original code - it passes 0, not
-1, as file descriptor parameter and because of that the mmap() calls would fail
on FreeBSD.
Linux is probably more permissive here.
So we need a patch for this.

> #elif HAVE_VIRTUALALLOC
>             c->lumMmx2FilterCode = VirtualAlloc(NULL, c->lumMmx2FilterCodeSize,
> MEM_COMMIT, PAGE_EXECUTE_READWRITE);
>             c->chrMmx2FilterCode = VirtualAlloc(NULL, c->chrMmx2FilterCodeSize,
> MEM_COMMIT, PAGE_EXECUTE_READWRITE);
> #else
>             c->lumMmx2FilterCode = av_malloc(c->lumMmx2FilterCodeSize);
>             c->chrMmx2FilterCode = av_malloc(c->chrMmx2FilterCodeSize);
> #endif
> 
>             if (!c->lumMmx2FilterCode || !c->chrMmx2FilterCode)
>                 goto fail;

The above check should also be fixed for the case when mmap is used.
mmap returns MAP_FAILED in case of failure and on FreeBSD MAP_FAILED != NULL.
Not sure about Linux, but we probably don't care - if only to report to upstream
to make their code cleaner.


Apologies again for not reporting these important issues.

-- 
Andriy Gapon