How to catch a memory leak?

[Roger Olofsson]

Dear Mailing List,

The application vlc (found in ports) when run on FBSD 5.x and 6.x 
behaves as if there's a memory leak somewhere hidden in it. This is 
appearing when starting vlc from shell playing a playlist and streaming 
video over udp to LAN like so:

vlc --loop playlist.m3u --sout 
'#transcode{acodec=mpga,ab=192}:std{access=udp,mux=,dst=239.255.255.255}'

The memory of the machine slowly gets eaten until the application ends 
with the message 'Killed' when the memory has run out.

I have adressed this on the vlc forum and I have begun glancing on the 
massive source code of this very complex application.

Kris Kennaway has pointed out that it is most likely an issue with threads.

My question is how do I track down a possible memory leak and would 
there be a tool to monitor (from outside the application) what process 
that allocates memory and how much?

Running it through gdb breaks it and attaching gdp to it while running 
breaks it.

Grateful for any pointers to get me in the right direction. Thanks In 
Advance.