amd64 process sizes

[Peter Wemm]

Ruslan Ermilov wrote:
> On Wed, Sep 05, 2007 at 07:50:49PM +1000, Peter Jeremy wrote:
>   
>> I've been comparing VSZ reported for similar processes on amd64 and
>> i386 and found that amd64 processes report as vastly larger than I
>> expected:
>>
>> i386  amd64
>> 1476   3572  getty
>> 6272  29184  xclock
>> 4784  28020  xdm
>> 6420  30000  xterm
>> 3828   9444  sendmail
>>
>> I did some further digging into the procfs map for xterm and found
>> that each of the amd64 shared objects has an additional mapping that
>> is 255 or 256 pages.  Once you remove those mappings, the sizes are
>> reasonably similar.  A typical set of mappings is:
>>
>> 0x800f60000 0x800fba000 61 0 0xffffff0027ac2540 r-x 36 18 0x0 COW NC vnode /usr/local/lib/libXaw8.so.8
>> 0x800fba000 0x800fbb000 1 0 0xffffff002765ed20 r-x 1 0 0x2180 COW NC vnode /usr/local/lib/libXaw8.so.8
>> 0x800fbb000 0x8010bb000 18 0 0xffffff0027ac2540 r-x 36 18 0x0 COW NC vnode /usr/local/lib/libXaw8.so.8 [*]
>> 0x8010bb000 0x8010cd000 18 0 0xffffff00274d2c40 rw- 1 0 0x2180 COW NC vnode /usr/local/lib/libXaw8.so.8
>>
>> 0x28284000 0x282d7000 15 0 0xc360ad68 r-x 9 6 0x0 COW NC vnode /usr/local/lib/libXaw8.so.8
>> 0x282d7000 0x282d8000 0 0 0xc3a4b210 r-x 1 0 0x2180 COW NC vnode /usr/local/lib/libXaw8.so.8
>> 0x282d8000 0x282df000 3 0 0xc3aef39c rwx 1 0 0x2180 COW NC vnode /usr/local/lib/libXaw8.so.8
>>
>> Could someone please explain the purpose of the mapping [*] and what
>> system resources they occupy.
>>
>>     
> These mappings are produced by rtld(1) when it maps shared objects.
> I took cat(1) here, and examined its libc.so's mappings on i386 and
> amd64.  What rtld(1) basically does is mmap's the PT_LOAD segments
> (see the Program Header in the "objdump -x /lib/libc.so.*" output).
> Since the size of the loadable segment is not necessaily a multiple
> of the page size, it zeroes out the rest of the last page of each
> loadable segment.  If the segment is read-only (like is the case
> for the "text" segment), it temporarily write-unprotects the last
> page, then zeroes its tail, then makes it read-only again.
>
> The first mapping you see is created by mapping a "text" segment
> of the library.  The last mapping is for the "data" segment.
> The second mapping is created by toggling the write protection
> of the page, and this mapping is 4096 bytes in size.
>
> The mysterious third mapping on amd64 is due to a gap between the
> virtual addresses of the two loadable segments (text and data) on
> amd64:
>
> Program Header:
>    LOAD off    0x0000000000000000 vaddr 0x0000000000000000 paddr 0x0000000000000000 align 2**20
>         filesz 0x00000000000d7051 memsz 0x00000000000d7051 flags r-x
>    LOAD off    0x00000000000d7060 vaddr 0x00000000001d7060 paddr 0x00000000001d7060 align 2**20
>         filesz 0x000000000001a010 memsz 0x0000000000032df8 flags rw-
>
> vend0 = rounddown(vaddr0, pagesize) + roundup(memsz0, pagesize) = 0x0 + 0xd8000 = 0xd8000
> gap = rounddown(vaddr1, pagesize) - vend0 = 0x1d7000 - 0xd8000 = 255 pages
>
> On i386 there's no gap:
>
> Program Header:
> LOAD off    0x00000000 vaddr 0x00000000 paddr 0x00000000 align 2**12
>      filesz 0x000ca5d2 memsz 0x000ca5d2 flags r-x
> LOAD off    0x000ca5e0 vaddr 0x000cb5e0 paddr 0x000cb5e0 align 2**12
>      filesz 0x000053f0 memsz 0x0001b404 flags rw-
>
> vend0 = rounddown(vaddr0, pagesize) + roundup(memsz0, pagesize) = 0x0 + 0xcb000 = 0xcb000
> gap = rounddown(vaddr1, pagesize) - vend0 = 0xcb000 - 0xcb000 = 0 pages
>
> Why amd64 linker does so, I don't know.  :-)
>
>
> Cheers,
>   
This is by design, at least from the binutils side of things.  There are 
two bugs.

1) binutils has the wrong max page size.  It is currently set at 1MB 
rather than 2MB.  I believe this is fixed in binutils sources that we 
have not imported yet.  This is currently harmless, but is less than 
useful once we have superpages support committed.

2) There is a bug in sys/kern/imgact_elf.c.  It assumes that the PT_LOAD 
sections are contiguous, which isn't a given.

Fixing #1 will give us future benefits.  Fixing #2 will reduce the 
virtual process size by eliminating the unnecessary mappings.  I'll take 
a look and see if I can fix it.  If somebody else wants to take a shot, 
be my guest.

We need to fix #2 because running linux binaries is affected.  Linux 
will be using the binutils layout.

-Peter