leaking lots of unreferenced inodes (pg_xlog files?)

[Palle Girgensohn]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello again!

I have now remounted the postgresql filesystem on a test server that
experiences the same problem. The production server is not remounted
yet, we're planning that in a weeks time approximately, but I though I
could gain som time by running the suggested procedure on the test box.

The base problem was this:

# df -h /pgsql ; du -hxs /pgsql
Filesystem     Size    Used   Avail Capacity  Mounted on
/dev/da2s1d    128G    101G     17G    86%    /pgsql
 82G	/pgsql

df says 101 GB used, but du only finds 82 GB, and fstat cannot find any
open files that are unreferenced in the file system. Stopping postgresql
does not help. It seems the OS is leaking inode references.

FreeBSD 9.1, postgresql 9.2.3 from port.

I ran the suggested commans (in attached diskspacecheck) before stopping
postgresql (before.log), after stopping postgresql but before unmount
/pgsql (before2.log), and then i unmounted /pgsql (had to run umount -f
/pgsql, and it took about 20 seconds). I did not enter single-user mode,
since I really did not have to this time (On the production server, the
disk is /usr, so that will require more shutting down...)

I've attach the logs here. Hope it helps!

The commands run in diskspaccheck are
#! /bin/sh
df -ih /pgsql
vmstat -z
vmstat -m
sysctl debug
fstat -f /pgsql

as suggested by Kirk.








Jeff Roberson skrev:
> On Mon, 6 May 2013, Kirk McKusick wrote:
> 
>>> Date: Sat, 04 May 2013 15:34:33 +0200 From: Palle Girgensohn
>>> <girgen at freebsd.org> To: Kirk McKusick <mckusick at mckusick.com> 
>>> Subject: Re: leaking lots of unreferenced inodes (pg_xlog
>>> files?), maybe after moving tables and indexes to tablespace on
>>> different volume Cc: freebsd-fs at freebsd.org, Jeff Roberson
>>> <jroberson at jroberson.net>
>>> 
>>> Hi,
>>> 
>>> Just a quick ping on this issue, it is still happening and we are
>>> slowly filling up the disk again. Is seems like we will have to
>>> plan a remount within a month given the current graphs. I will be
>>> back with info once we have remounted it, running you suggested
>>> scripts before and after.
>>> 
>>> Is there anything else I can do to get more debug information?
>>> 
>>> Regards, Palle
>> 
>> In addition to the things that I mentioned previously please also
>> run `vmstat -z' before and after the unmount/mount.
>> 
>> Jeff, do you have any other things that you think he should run?
>> 
>> Kirk
>> 
> 
> Remounting fixes it?  Does it require a fsck?  I don't have enough 
> context to comment here.  The vmstat will tell us if it's a leak in
> a memory reference but that should blow up memory before is saturates
> a volume unless the volume is very small.  How big is the volume?
> How much memory do you have?  The inodes show up as used in df -i?
> How many more than find sees?  What is kern.maxvnodes and
> vfs.numvnodes?
> 
> Jeff
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJRpzB2AAoJEIhV+7FrxBJDVIYIAK8a2zAx6pUlDn1CDj1AlSb3
jvNRh/YgTUJBj0JSkh93uRqi8A6sy1gGz6PHqOhUE4uN+nNXCIqkYvPkLJ7C8zRW
O6LCJ9nSMud6woGpeBRz8V3bu2RbHZMOATZ2f1dyzFPOyB8nU02zsoK3yOojUsWk
PpCGIpSZafKfP1A98wKNxFBoo59+HxePqbAEQeP4Cl5+/Hudnj6hwBdB0DsZkhkU
4vSVEzQ360iiQDFTurxIv+CARvtrnhYC+iFxPsQftevEbhlRH5z9YsUZkR5YVms/
h+Y551yqB4NrsRqcw0qQOf9lNb10PK74z7sCrizsIKFonYkXGQSv16AXWXnG95I=
=LvOT
-----END PGP SIGNATURE-----
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: before.log
URL: <http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20130530/28e53c20/attachment-0002.ksh>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: before2.log
URL: <http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20130530/28e53c20/attachment-0003.ksh>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: after.log
Type: application/octet-stream
Size: 26121 bytes
Desc: not available
URL: <http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20130530/28e53c20/attachment-0001.obj>