file system help needed

[Paul Schenkeveld]

Hi all,

I just saw this thread, hence this late reaction.  Hopefully I'm 
late and you got your system in shape again but in case you still
didn't find the culprit, here are my $0.02.

First, did you ever "ktrace" a process and remove the ktrace.out
file?  In that case, "ktrace -C" will instruct the kernel to stop
tracing and release all ktrace.out files, including those that you
already removed.

Then, I'd check for files that were removed but are still open by
some process and continue to occupy diskspace.

To check this: compare the output of "du -skx /" to the value
reported under Used in the output of "df -l /".

If the two are (almost) the same, all space is occupied by files
that are still around in some directory, in that case you really
have to look though all directories of the root filesystem to find
what's there but should not be there (a common culprit is a regular
file in /dev, possibly with a name looking like the name of a tape
drive or even /dev/null).

If however these two numbers are very different you've got removed
files still occupying disk space because they are open.  In that case
you can either reboot the machine compelete if possible (make sure
/sbin and /bin are back first otherwise you even don't have a mount
command anymore and got yourself a chicken and egg situation), or
stop processes one by one (and restart them afterwards) until df
suddenly shows free space again (hopefully you've not enabled soft
updates on / in which case recovery of free space will not show up
in df until up to half a minute or so after stopping the process
which kept the file open).

Or you can take a more deterministic approach by looking au the output
of fstat.  The MOUNT column will show / for all inodes (files) open
that are on the root filesystem.  Concentrate on regular files (lines
with a '-' as the first character of the MODE column) and look at the
SZ|DV column.  Huge numbers here indicate huge files.  Skip lines
with "text" in the FD column and try to find suspicious files using
"find / -xdev -inum <inum>" using the inode number from the INUM
column.  This should reveal processes keeping deleted files open,
kill those culprits and if they served a purpose, restart them
again.

Regards,

Paul Schenkeveld, Consultant
PSconsult ICT Services BV

On Tue, Mar 25, 2003 at 06:14:59PM +0100, Arie J. Gerszt wrote:
> Hi Everybody
> 
> I have a huge problem as you see ...
> 
> Filesystem  1K-blocks     Used    Avail Capacity  Mounted on
> /dev/ad0s1a     99183    96309    -5060   106%    /
> /dev/ad0s1e   5458605  2866595  2155322    57%    /usr
> /dev/ad0s1g  17876344  1085421 15360816     7%    /usr/www
> /dev/ad0s1f   4465853   212592  3895993     5%    /var
> procfs              4        4        0   100%    /proc
> 
> This server is a production server and I can't add any dns zonefiles or
> passwords,
> because they sit in /etc which is full, evidently.
> 
> What can I do to solve that rather quickly? The disk ad0 has free space:
> 
> caramba# fdisk /dev/ad0
> ******* Working on device /dev/ad0 *******
> parameters extracted from in-core disklabel are:
> cylinders=3649 heads=255 sectors/track=63 (16065 blks/cyl)
> 
> Figures below won't work with BIOS for partitions not in cyl 1
> parameters to be used for BIOS calculations are:
> cylinders=3649 heads=255 sectors/track=63 (16065 blks/cyl)
> 
> Media sector size is 512
> Warning: BIOS sector numbering starts with sector 1
> Information from DOS bootblock is:
> The data for partition 1 is:
> sysid 165,(FreeBSD/NetBSD/386BSD)
>     start 63, size 58621122 (28623 Meg), flag 80 (active)
>         beg: cyl 0/ head 1/ sector 1;
>         end: cyl 1023/ head 254/ sector 63
> The data for partition 2 is:
> <UNUSED>
> The data for partition 3 is:
> <UNUSED>
> The data for partition 4 is:
> <UNUSED>
> caramba#
> 
> 
> --> but i am not sure what to do, because I can't have any long downtime,
> understandibily.
> Thanks for help,
> 
> Arie
> 
> 
> 
> To Unsubscribe: send mail to majordomo at FreeBSD.org
> with "unsubscribe freebsd-isp" in the body of the message