Strange ZFS problem, filesystem claims to be full when clearly not full

[Torbjorn Kristoffersen]

I'm at a complete loss here. I shut down the jail completely, and I am
watching the jail's ZFS filesystem grow as we speak.  No process is using
it.   It only grows in "df" and "zfs list", I can't find any files that are
growing.  I have to re-set the quota to be higher and higher to accommodate
the space.

On Wed, Sep 29, 2010 at 10:46 PM, Torbjorn Kristoffersen <
torbjoern at gmail.com> wrote:

> Hi Jeremy.
>
> 1) I checked now, and found nothing extraordinary. Just processes that have
> been running for a long while, such as screen, cron, sshd, bash, irssi,
> syslogd, etc.
>
> 2) No compression used on this zfs filesystem (or any of the others).
>
> I completedly stopped the jail now, and removed some of the directories
> with the most data in them, but to no avail.
>
>
> On Wed, Sep 29, 2010 at 9:25 PM, Jeremy Chadwick <freebsd at jdc.parodius.com
> > wrote:
>
>> On Wed, Sep 29, 2010 at 08:46:38PM +0200, Torbjorn Kristoffersen wrote:
>> > I have a ZFS "tank" called tpool, the server runs a couple of jails
>> (each
>> > with a zfs filesystem).  There is a problem with one of these
>> filesystems.
>> > First, its disk usage as shown in ``df -h'':
>> > ...
>> > tpool/rb.org      100G     95G    4.6G    95%    /jails/rb.org
>> > ...
>> >
>> > The command ``zfs list'' shows the same:
>> > ..
>> > tpool/rb.org    95.4G  4.56G  95.4G  /jails/rb.org
>> > ..
>> >
>> > However, there is a very mysterious problem somewhere.
>> > Something inside this jail is eating diskspace, but we can't find any
>> > directories that is actually taking the diskspace. We first suspected
>> either
>> > fetchmail or spamassassin of causing a lot of space to be used, since
>> some
>> > of their directories were huge. (These were later deleted, and which is
>> why
>> > you see that 4.6GB is now available, before that 0GB was available).
>> >
>> > However, we can't find *any trace* of an actual directory or file that
>> is
>> > taking all the spac.e
>> >
>> > Take this for instance:
>> >
>> > outsidejail# du -sh rb.org
>> >  43G    rb.org
>> >
>> > How can this be?  df and zfs are showing that the entire drive is nearly
>> > full, yet I can't find any directory that is actually taking all this
>> space.
>> >  I've carefully looked through every single directory within the jail
>> trying
>> > to find something that's taking all that space, but to no avail.
>> >
>> > ----
>> > My system stats:
>> > # uname -a
>> > FreeBSD grim 8.1-RELEASE FreeBSD 8.1-RELEASE #0: Mon Jul 19 02:36:49 UTC
>> > 2010     root at mason.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  amd64
>> > # zpool get version tpool
>> > NAME   PROPERTY  VALUE    SOURCE
>> > tpool  version   14       default
>> > # zpool status
>> >   pool: tpool
>> >  state: ONLINE
>> >  scrub: none requested
>> > config:
>> >
>> >         NAME        STATE     READ WRITE CKSUM
>> >         tpool       ONLINE       0     0     0
>> >           mirror    ONLINE       0     0     0
>> >             ad4s1d  ONLINE       0     0     0
>> >             ad6s1d  ONLINE       0     0     0
>> >
>> > errors: No known data errors
>> >
>> > [ Note that I've also done a scrub recently ]
>>
>> 1) Have you checked using fstat to ensure that no file descriptors
>> remain open on any of your ZFS filesystems (not pools)?
>>
>> 2) Are you using compression on any of your ZFS filesystems?
>>
>> --
>> | Jeremy Chadwick                                   jdc at parodius.com |
>> | Parodius Networking                       http://www.parodius.com/ |
>> | UNIX Systems Administrator                  Mountain View, CA, USA |
>> | Making life hard for others since 1977.              PGP: 4BD6C0CB |
>>
>>
>