Memory leak on 5.1-RELEASE?

Jorn Argelo jorn at wcborstel.nl
Sat Jan 31 15:30:41 PST 2004


How large are you MySQL databases? I suppose a large and intensly-used 
database eats away quite some RAM. What services is the inetd giving? 

You can disable some of your shells by editing the /etc/ttys file, saves some 
RAM as well, but it won't really make a difference when you have such a large 
amount of RAM in your machine. 

You should really try to disable some unnessesairy services. If it isn't a 
production box, then I'd say you kill the services one at a time and bring 
them back up again, so you can see what is eating away all your RAM. It could 
be that top is just giving wrong information though, but I can't be certain 
of that. If your machine is just running the required processes and top still 
says you're using much RAM, you can try to compile the kernel / world by 
yourself. This can improve performance and reducing the required RAM needed 
by the kernel (support for several RAID and SCSI controllers which you don't 
need). Of course, this isn't possible if it's a production box.

Cheers, 

Jorn.

On Saturday 31 January 2004 23:04, you wrote:
> Here's the top output from this morning:
>
> Mem: 1300M Active, 292M Inact, 295M Wired, 104M Cache, 112M Buf, 5116K Free
> Swap: 1024M Total, 8036K Used, 1016M Free
>
> At this time there are ~130 processes, whereas in the previous one
> there were ~76, with more memory marked as "Active". vmstat says this:
>
>  procs      memory      page                    disks     faults      cpu
>  r b w     avm    fre  flt  re  pi  po  fr  sr da0 ac0   in   sy  cs us sy
> id 0 0 1  848048 100880  937 113  23   0 1503 248   0   0 2678    0 8211  2
>  4 95 0 0 1  830192 110484  340   4   1   0 2719   0   1   0 2574    0 5917
>  1  5 93
>
> avm is supposed to be the number of active virtual pages. The page size
> being 4096, according to hw.pagesize, would actually come up to around
> 3.4GB. I'm not sure what this number refers to - maybe it's memory + swap?
>
> The whole reason I'm bringing this up with the list is that, ultimately,
> this is causing the machine to crash, after running out of swap. If I
> could identify where this Active memory has gone, I may be able to recover
> the "lost" memory and prevent the server from swapping itself to death.
>
> Here's the ps -ax output - note that nfsd is running, but it's not being
> used. It does have an NFS client mounted, but from "vmstat -m", it isn't
> using any memory at all (and it would be in Buf I think?)
>
>   PID  TT  STAT      TIME COMMAND
>     0  ??  DLs    0:00.82  (swapper)
>     0  ??  ZW     0:00.00  (httpd)
>     1  ??  ILs    0:00.08 /sbin/init --
>     2  ??  DL     0:13.59  (g_event)
>     3  ??  DL     3:41.95  (g_up)
>     4  ??  DL     4:56.15  (g_down)
>     5  ??  DL     1:41.88  (pagedaemon)
>     6  ??  DL     0:00.00  (vmdaemon)
>     7  ??  DL    16:25.54  (pagezero)
>     8  ??  DL     2:07.11  (bufdaemon)
>     9  ??  DL     0:01.51  (vnlru)
>    10  ??  DL     0:00.00  (ktrace)
>    11  ??  RL     0:00.00  (idle: cpu3)
>    12  ??  RL   5037:43.41  (idle: cpu2)
>    13  ??  RL     0:00.00  (idle: cpu1)
>    14  ??  RL   4281:37.99  (idle: cpu0)
>    15  ??  WL   127:48.20  (swi1: net)
>    16  ??  WL     6:02.49  (swi7: tty:sio clock)
>    18  ??  DL     1:26.21  (random)
>    20  ??  WL     1:39.12  (swi3: cambio)
>    23  ??  WL     0:00.00  (irq14: ata0)
>    26  ??  DL     0:00.02  (usb0)
>    27  ??  DL     0:00.00  (usbtask)
>    30  ??  WL     2:13.96  (irq16: ahc0)
>    31  ??  WL     0:00.00  (irq17: ahc1)
>    32  ??  WL    58:17.09  (irq18: em0)
>    33  ??  WL     0:00.00  (irq1: atkbd0)
>    34  ??  WL     0:00.00  (irq6: fdc0)
>    35  ??  WL     0:00.00  (swi0: tty:sio)
>    40  ??  DL    13:22.17  (syncer)
>    41  ??  IL     3:06.26  (nfsiod 0)
>    42  ??  IL     2:47.50  (nfsiod 1)
>    43  ??  IL     2:28.19  (nfsiod 2)
>    44  ??  IL     3:37.52  (nfsiod 3)
>   148  ??  Is     0:00.00 adjkerntz -i
>   257  ??  Ss     0:02.08 /usr/sbin/syslogd -s
>   352  ??  Ss     0:00.25 /usr/sbin/rpcbind
>   375  ??  Is     0:00.01 /usr/sbin/mountd -r
>   377  ??  Is     0:00.02 nfsd: master (nfsd)
>   380  ??  I      0:00.00 nfsd: server (nfsd)
>   381  ??  I      0:00.00 nfsd: server (nfsd)
>   382  ??  I      0:00.00 nfsd: server (nfsd)
>   383  ??  I      0:00.00 nfsd: server (nfsd)
>   394  ??  Ss     0:00.35 /usr/sbin/usbd
>   443  ??  Is     0:02.00 /usr/sbin/sshd -u0
>   449  ??  Ss     0:06.17 sendmail: accepting connections (sendmail)
>   452  ??  Is     0:00.13 sendmail: Queue runner at 00:30:00 for
> /var/spool/client 466  ??  Is     0:01.25 /usr/sbin/cron
>   829  ??  Ss     0:02.75 proftpd: (accepting connections) (proftpd)
>   855  ??  Is     0:00.00 /usr/sbin/inetd -wW
> 21727  ??  Ss     2:11.11 /usr/local/bin/httpd -f
> /usr/local/www/server/conf/ht 31784  ??  S      0:00.23
> /usr/local/bin/httpd -f /usr/local/www/server/conf/ht 52298  ??  S     
> 0:00.13 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht 52989  ??  S 
>     0:00.18 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht 53224  ??
>  S      0:00.11 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht 54180
>  ??  S      0:00.18 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
> 54282  ??  S      0:00.13 /usr/local/bin/httpd -f
> /usr/local/www/server/conf/ht 54310  ??  S      0:00.13
> /usr/local/bin/httpd -f /usr/local/www/server/conf/ht 55563  ??  S     
> 0:00.11 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht 55970  ??  S 
>     0:00.16 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht 56261  ??
>  S      0:00.13 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht 56369
>  ??  S      0:00.10 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
> 56422  ??  S      0:00.04 /usr/local/bin/httpd -f
> /usr/local/www/server/conf/ht 56424  ??  S      0:00.18
> /usr/local/bin/httpd -f /usr/local/www/server/conf/ht 56465  ??  S     
> 0:00.11 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht 56491  ??  S 
>     0:00.17 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht 56523  ??
>  S      0:00.12 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht 56575
>  ??  Rs     0:00.07 sshd: toor at ttyp0 (sshd)
> 56606  ??  S      0:00.33 /usr/local/bin/httpd -f
> /usr/local/www/server/conf/ht 56617  ??  S      0:00.15
> /usr/local/bin/httpd -f /usr/local/www/server/conf/ht 56729  ??  I     
> 0:00.13 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht 56763  ??  S 
>     0:00.11 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht 56768  ??
>  S      0:00.09 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht 56779
>  ??  S      0:00.09 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
> 56786  ??  S      0:00.05 /usr/local/bin/httpd -f
> /usr/local/www/server/conf/ht 56788  ??  S      0:00.00
> /usr/local/bin/httpd -f /usr/local/www/server/conf/ht 56789  ??  S     
> 0:00.12 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht 56790  ??  S 
>     0:00.08 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht 56800  ??
>  S      0:00.09 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht 56809
>  ??  I      0:00.01 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
> 56811  ??  S      0:00.13 /usr/local/bin/httpd -f
> /usr/local/www/server/conf/ht 56840  ??  S      0:00.15
> /usr/local/bin/httpd -f /usr/local/www/server/conf/ht 56843  ??  S     
> 0:00.04 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht 56845  ??  S 
>     0:00.03 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht 56846  ??
>  S      0:00.02 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht 56852
>  ??  S      0:00.02 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
> 56853  ??  S      0:00.05 /usr/local/bin/httpd -f
> /usr/local/www/server/conf/ht 56864  ??  S      0:00.11
> /usr/local/bin/httpd -f /usr/local/www/server/conf/ht 56865  ??  S     
> 0:00.17 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht 56866  ??  S 
>     0:00.06 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht 56869  ??
>  S      0:00.06 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht 56870
>  ??  S      0:00.06 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
> 56871  ??  S      0:00.05 /usr/local/bin/httpd -f
> /usr/local/www/server/conf/ht 56872  ??  S      0:00.09
> /usr/local/bin/httpd -f /usr/local/www/server/conf/ht 56873  ??  S     
> 0:00.03 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht 56874  ??  S 
>     0:00.04 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht 56875  ??
>  S      0:00.07 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht 56876
>  ??  S      0:00.03 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
> 56877  ??  S      0:00.03 /usr/local/bin/httpd -f
> /usr/local/www/server/conf/ht 56878  ??  S      0:00.04
> /usr/local/bin/httpd -f /usr/local/www/server/conf/ht 56879  ??  S     
> 0:00.07 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht 56880  ??  S 
>     0:00.07 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht 56881  ??
>  S      0:00.06 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht 56882
>  ??  S      0:00.04 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
> 56883  ??  S      0:00.05 /usr/local/bin/httpd -f
> /usr/local/www/server/conf/ht 56884  ??  S      0:00.03
> /usr/local/bin/httpd -f /usr/local/www/server/conf/ht 56885  ??  S     
> 0:00.03 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht 56886  ??  S 
>     0:00.04 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht 56887  ??
>  S      0:00.12 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht 56888
>  ??  S      0:00.02 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
> 56889  ??  S      0:00.03 /usr/local/bin/httpd -f
> /usr/local/www/server/conf/ht 56890  ??  S      0:00.03
> /usr/local/bin/httpd -f /usr/local/www/server/conf/ht 56891  ??  S     
> 0:00.03 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht 56892  ??  S 
>     0:00.02 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht 56893  ??
>  S      0:00.04 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht 56894
>  ??  S      0:00.03 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht
> 56896  ??  S      0:00.00 /usr/local/bin/httpd -f
> /usr/local/www/server/conf/ht 56897  ??  S      0:00.02
> /usr/local/bin/httpd -f /usr/local/www/server/conf/ht 56898  ??  S     
> 0:00.03 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht 56900  ??  S 
>     0:00.03 /usr/local/bin/httpd -f /usr/local/www/server/conf/ht 56577  p0
>  Ss     0:00.02 -sh (sh)
> 56904  p0  R+     0:00.00 ps -ax
>  3574  v0  Is+    0:00.00 /usr/libexec/getty Pc ttyv0
>   868  v1  Is+    0:00.00 /usr/libexec/getty Pc ttyv1
>   869  v2  Is+    0:00.00 /usr/libexec/getty Pc ttyv2
>   870  v3  Is+    0:00.00 /usr/libexec/getty Pc ttyv3
>   871  v4  Is+    0:00.00 /usr/libexec/getty Pc ttyv4
>   872  v5  Is+    0:00.00 /usr/libexec/getty Pc ttyv5
>   873  v6  Is+    0:00.00 /usr/libexec/getty Pc ttyv6
>   874  v7  Is+    0:00.00 /usr/libexec/getty Pc ttyv7
>   520 con- I      0:02.81 /usr/local/sbin/snmpd
>   534 con- I      0:00.01 /bin/sh /usr/local/bin/mysqld_safe
> --datadir=/usr/loc 566 con- S    138:26.09 /usr/local/libexec/mysqld
> --basedir=/usr/local --data
>
> According to top and ps, mysql is only using around 59M VSZ.
>
> On Sun, 1 Feb 2004, Jorn Argelo wrote:
> > Are you sure that there isn't anything else running? Why don't  you give
> > us an ps -ax output? I don't think there's a memory leak in 5.1, since
> > I've seen running 5.1 just fine on a PE2650 with 2 GB RAM. You shouldn't
> > rely on top too much acually. Vmstat is a better program when looking at
> > memory.
> >
> > Cheers,
> >
> > Jorn
> > ----- Original Message -----
> > From: "dpk" <dpk at dpk.net>
> > To: <questions at freebsd.org>
> > Sent: Saturday, January 31, 2004 1:10 AM
> > Subject: Memory leak on 5.1-RELEASE?
> >
> > > (I'm not a member of the list; please Cc me on any replies.)
> > >
> > > We're running Apache 1.3.28 on a 5.1-RELEASE machine. It's a Dell PE
> > > 2650 w/ 2GB RAM. The site contains a lot of large files
> > > (multi-megabyte) - otherwise there's nothing unusual running.
> > >
> > > The Active memory use, according to top, seems rather high:
> > >
> > > last pid: 21487;  load averages:  0.19,  0.33,  0.32    up 2+16:45:20
> >
> > 15:52:21
> >
> > > 76 processes:  1 running, 75 sleeping
> > > CPU states:  0.5% user,  0.0% nice,  4.0% system,  1.4% interrupt,
> > > 94.2%
> >
> > idle
> >
> > > Mem: 1413M Active, 187M Inact, 299M Wired, 93M Cache, 112M Buf, 2632K
> > > Free Swap: 1024M Total, 21M Used, 1003M Free, 2% Inuse
> > >
> > > We can't seem to get the Active number down much, even after stopping
> > > Apache it still stays around 1100M. There's no shared memory in use,
> > > and nothing in vmstat -m seems to indicate where the "missing" memory
> > > is. top, sorting by "size", does not indicate anything unusual either.
> > >
> > > sysctl vm.vmtotal says:
> > >
> > > vm.vmtotal:
> > > System wide totals computed every five seconds: (values in kilobytes)
> > > ===============================================
> > > Processes:              (RUNQ: 1 Disk Wait: 0 Page Wait: 0 Sleep: 76)
> > > Virtual Memory:         (Total: 8172K, Active 636472K)
> > > Real Memory:            (Total: 2051312K Active 389176K)
> > > Shared Virtual Memory:  (Total: 16436K Active: 11760K)
> > > Shared Real Memory:     (Total: 6004K Active: 4436K)
> > > Free Memory Pages:      79228K
> > >
> > > whereas on other servers, the Real Memory "Active" number seems to
> > > match the one found in top, on this one it is about 1GB lower.
> > >
> > > A similar machine running Apache on 5.1-R, generally serving smaller
> > > files, has the same problem in a smaller scale (about 640M even when
> > > Apache is stopped).
> > >
> > > Are there any other data that I should send to help diagnose this
> > > problem, or any programs I can run to try and track this stray memory
> > > use down?
> > >
> > > - dpk
> > > _______________________________________________
> > > freebsd-questions at freebsd.org mailing list
> > > http://lists.freebsd.org/mailman/listinfo/freebsd-questions
> > > To unsubscribe, send any mail to
> >
> > "freebsd-questions-unsubscribe at freebsd.org"



More information about the freebsd-questions mailing list