collecting pv entries -- suggest increasing PMAP_SHPGPERPROC
Chuck Swiger
cswiger at mac.com
Tue Oct 28 11:27:58 PDT 2008
On Oct 28, 2008, at 9:49 AM, Francis Dubé wrote:
>>> Here's an example of top's output regarding our httpd process :
>>> 54326 apache 1 96 0 156M 13108K select 1 0:00
>>> 0.15% httpd
>>> 54952 apache 1 96 0 156M 12684K select 1 0:00
>>> 0.10% httpd
>>> 52343 apache 1 4 0 155M 12280K select 0 0:01
>>> 0.10% httpd
>>>
>>> Most of our page are in HTML with a LOT of images. Few PHP pages,
>>> very light PHP processing.
>>>
>>> 156M x 450 process = way more RAM than what we have (same for
>>> RES). Concretely, how must I interpret these results?
First, your Apache children are huge, at least for FreeBSD. :-)
Also, they are mostly paged out, which suggests your system is under
significant VM pressure, but the vmstat output would be helpful to
confirm.
>> It's as I expected -- you don't understand the difference between
>> SIZE (SZ) and RES (RSS). The simple version:
>>
>> SIZE == amount of memory that's shared across all processes on the
>> machine, e.g. shared libraries. It doesn't mean "156MB is being
>> taken
>> up per process".
SIZE == the amount of VM address space allocated by the process.
It includes things shared (copy-on-write) between many processes like
the shared libraries; it also includes memory-mapped files
(including .so's like apache modules being loaded into the process),
VM allocated but not yet used by malloc()/brk(), the stack, and so
forth.
>> RES == amount of memory that's specifically allocated to that
>> individual
>> process. The three httpd processes above are taking up a total of
>> ~38MBytes of memory (13108K + 12684K + 12280K).
RES == the amount of process VM that is resident in actual physical
RAM; the rest of the process is paged out to the swapfile or
filesystem for memory-mapped files.
> As I said, even with RES the numbers dont seems to have any sense.
>
> Let's say 12500K x 450 = ~5500MBytes. Considering there's a lot of
> process other than Apache running on the server...there's something
> wrong. Is there something shared in RES too ?
Yep. Quite probably a lot, but the amount of memory which is specific
to just that process is not easily found from FreeBSD's top,
regrettably.
For the sake of example, and because the same explanation applies
pretty closly to FreeBSD, consider an httpd running on a MacOSX
system. Here's top output, which includes columns "RPRVT" for
"resident memory used by just this process", "RSHRD" which is
"resident, shared with other processes", "RSIZE" which is FreeBSD's
"RES", and "VSIZE", which is FreeBSD's "SIZE":
Processes: 136 total, 4 running, 132 sleeping... 215 threads
11:06:40
Load Avg: 1.71, 1.66, 1.62 CPU usage: 12.5% user, 59.7% sys,
27.8% idle
SharedLibs: num = 141, resident = 18.3M code, 2.92M data, 6.40M
LinkEdit
MemRegions: num = 10360, resident = 101M + 5.91M private, 159M shared
PhysMem: 159M wired, 252M active, 99.0M inactive, 510M used, 1.50G
free
VM: 7.16G + 88.8M 1378510(0) pageins, 88743(0) pageouts
PID COMMAND %CPU TIME #TH #PRTS #MREGS RPRVT RSHRD
RSIZE VSIZE
2868 httpd 0.0% 43:21.28 1 12 92 1.82M 144M
72.9M 169M
2869 httpd 0.0% 46:29.45 1 12 92 1.95M 144M
73.2M 169M
2870 httpd 0.0% 46:55.84 1 12 92 1.89M 144M
73.0M 169M
...and the vmmap command, documented here:
http://developer.apple.com/documentation/Darwin/Reference/ManPages/man1/vmmap.1.html
...provides detailed info about a single process' VM usage:
# vmmap 2870
Virtual Memory Map of process 2870 (httpd)
Output report format: 2.0
==== Non-writable regions for process 2870
__PAGEZERO 00000000-00001000 [ 4K] ---/--- SM=NUL /usr/
sbin/httpd
__TEXT 00001000-00050000 [ 316K] r-x/rwx SM=COW /usr/
sbin/httpd
__LINKEDIT 0005a000-00065000 [ 44K] r--/rwx SM=COW /usr/
sbin/httpd
__TEXT 00065000-00068000 [ 12K] r-x/rwx SM=COW /usr/
libexec/httpd/mod_log_config.so
__LINKEDIT 00069000-0006a000 [ 4K] r--/rwx SM=COW /usr/
libexec/httpd/mod_log_config.so
__TEXT 0006a000-0006c000 [ 8K] r-x/rwx SM=COW /usr/
libexec/httpd/mod_mime.so
__LINKEDIT 0006d000-0006e000 [ 4K] r--/rwx SM=COW /usr/
libexec/httpd/mod_mime.so
[ ... ]
__DATA a1a0e000-a1a20000 [ 72K] r--/r-- SM=COW /usr/
lib/libcrypto.0.9.7.dylib
__DATA a1a20000-a1a23000 [ 12K] r--/r-- SM=COW /usr/
lib/libcrypto.0.9.7.dylib
__DATA a4f2c000-a4f2f000 [ 12K] r--/r-- SM=COW /usr/
lib/libssl.0.9.7.dylib
__DATA a7233000-a7235000 [ 8K] r--/r-- SM=NUL /
System/Library/Perl/lib/5.8/libperl.dylib
system fffec000-fffef000 [ 12K] ---/rwx SM=NUL
commpage [libobjc.A.dylib]
system fffef000-ffff0000 [ 4K] r-x/rwx SM=COW
commpage [libobjc.A.dylib]
system ffff8000-ffffa000 [ 8K] r--/r-- SM=SHM
commpage [libSystem.B.dylib]
==== Writable regions for process 2870
__DATA 00050000-00059000 [ 36K] rw-/rwx SM=COW /usr/
sbin/httpd
__DATA 00059000-0005a000 [ 4K] rw-/rwx SM=COW /usr/
sbin/httpd
__DATA 00068000-00069000 [ 4K] rw-/rwx SM=COW /usr/
libexec/httpd/mod_log_config.so
__DATA 0006c000-0006d000 [ 4K] rw-/rwx SM=COW /usr/
libexec/httpd/mod_mime.so
[ ... ]
__DATA a0a3a000-a0a4f000 [ 84K] rw-/rw- SM=COW /usr/
lib/libobjc.A.dylib
__OBJC a0a4f000-a0a50000 [ 4K] rw-/rw- SM=COW /usr/
lib/libobjc.A.dylib
__DATA a0b70000-a0b71000 [ 4K] rw-/rw- SM=COW /usr/
lib/libauto.dylib
__DATA a1425000-a1426000 [ 4K] rw-/rw- SM=COW /usr/
lib/libgcc_s.1.dylib
__DATA a7229000-a7233000 [ 40K] rw-/rw- SM=COW /
System/Library/Perl/lib/5.8/libperl.dylib
Stack bf800000-c0000000 [ 8192K] rw-/rwx SM=COW
thread 0
==== Legend
SM=sharing mode:
COW=copy_on_write PRV=private NUL=empty ALI=aliased
SHM=shared ZER=zero_filled S/A=shared_alias
==== Summary for process 2870
ReadOnly portion of Libraries: Total=18840KB resident=10976KB(58%)
swapped_out_or_unallocated=7864KB(42%)
Writable regions: Total=51100KB written=548KB(1%)
resident=27276KB(53%) swapped_out=0KB(0%) unallocated=23824KB(47%)
REGION TYPE [ VIRTUAL]
=========== [ =======]
MALLOC [ 42400K]
Stack [ 8192K]
VM_ALLOCATE ? [ 364K]
__DATA [ 1048K]
__LINKEDIT [ 4352K]
__OBJC [ 4K]
__PAGEZERO [ 4K]
__TEXT [ 14488K]
mapped file [ 120368K]
shared memory [ 4K]
system [ 24K]
Note that you can obtain somewhat similar information under FreeBSD
using the sysutils/pmap port:
PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU
COMMAND
40037 www 1 4 0 28820K 11532K accept 1:10 0.00%
httpd
# pmap 40037
40037: /usr/local/sbin/httpd
Address Kbytes RSS Shared Priv Mode Mapped File
08048000 280 208 280 - r-x /usr/local/sbin/httpd
0808E000 8 8 - 8 rw- /usr/local/sbin/httpd
08090000 16 - - - rw- [swap pager]
08094000 976 - - - rwx [swap pager]
08188000 2420 - - - rwx [swap pager]
083E5000 6968 - - - rwx [swap pager]
2808E000 144 92 144 - r-x /libexec/ld-elf.so.1
280B2000 8 4 8 - rw- /libexec/ld-elf.so.1
280B4000 20 - - - rw- [swap pager]
280B9000 32 - - - rwx [swap pager]
280C1000 64 16 64 - r-x /lib/libz.so.3
[ ... ]
2920F000 4 0 4 - r-x /usr/local/lib/php/
20060613/dom.so
29210000 16 0 16 - rwx /usr/local/lib/php/
20060613/dom.so
29214000 60 - - - rwx [swap pager]
29225000 16 - - - rwx [swap pager]
BFBE0000 128 - - - rwx [swap pager]
-------- ------- ------- ------- -------
Total Kb 28740 3996 17384 456
...and the pmap manpage talks about how to understand this:
" To calculate the amount of memory a group of processes is using:
- for each process of the same binary, count the private
memory
used for n-1 processes.
- for each binary, count the total resident size once.
For example, using the pmap data above, we can calculate that 20
bash
binaries would used
452Kbytes * 19 + 1416Kbytes = 8588 + 1416 Kbytes
= 8588Kbytes"
Regards,
--
-Chuck
More information about the freebsd-questions
mailing list