Strange ZFS problem, filesystem claims to be full when clearly not full

[Alexander Leidinger]

Quoting Jeremy Chadwick <freebsd at jdc.parodius.com> (from Wed, 29 Sep  
2010 15:15:49 -0700):

> On Thu, Sep 30, 2010 at 12:11:09AM +0200, Torbjorn Kristoffersen wrote:
>> I'm at a complete loss here. I shut down the jail completely, and I am
>> watching the jail's ZFS filesystem grow as we speak.  No process is using
>> it.   It only grows in "df" and "zfs list", I can't find any files that are
>> growing.  I have to re-set the quota to be higher and higher to accommodate
>> the space.
>>
>> On Wed, Sep 29, 2010 at 10:46 PM, Torbjorn Kristoffersen <
>> torbjoern at gmail.com> wrote:
>>
>> > Hi Jeremy.
>> >
>> > 1) I checked now, and found nothing extraordinary. Just processes  
>> that have
>> > been running for a long while, such as screen, cron, sshd, bash, irssi,
>> > syslogd, etc.
>> >
>> > 2) No compression used on this zfs filesystem (or any of the others).
>> >
>> > I completedly stopped the jail now, and removed some of the directories
>> > with the most data in them, but to no avail.
>> >
>> >
>> > On Wed, Sep 29, 2010 at 9:25 PM, Jeremy Chadwick <freebsd at jdc.parodius.com
>> > > wrote:
>> >
>> >> On Wed, Sep 29, 2010 at 08:46:38PM +0200, Torbjorn Kristoffersen wrote:
>> >> > I have a ZFS "tank" called tpool, the server runs a couple of jails
>> >> (each
>> >> > with a zfs filesystem).  There is a problem with one of these
>> >> filesystems.
>> >> > First, its disk usage as shown in ``df -h'':
>> >> > ...
>> >> > tpool/rb.org      100G     95G    4.6G    95%    /jails/rb.org
>> >> > ...
>> >> >
>> >> > The command ``zfs list'' shows the same:
>> >> > ..
>> >> > tpool/rb.org    95.4G  4.56G  95.4G  /jails/rb.org
>> >> > ..
>> >> >
>> >> > However, there is a very mysterious problem somewhere.
>> >> > Something inside this jail is eating diskspace, but we can't find any
>> >> > directories that is actually taking the diskspace. We first suspected
>> >> either
>> >> > fetchmail or spamassassin of causing a lot of space to be used, since
>> >> some
>> >> > of their directories were huge. (These were later deleted, and which is
>> >> why
>> >> > you see that 4.6GB is now available, before that 0GB was available).
>> >> >
>> >> > However, we can't find *any trace* of an actual directory or file that
>> >> is
>> >> > taking all the spac.e
>> >> >
>> >> > Take this for instance:
>> >> >
>> >> > outsidejail# du -sh rb.org
>> >> >  43G    rb.org
>> >> >
>> >> > How can this be?  df and zfs are showing that the entire drive  
>> is nearly
>> >> > full, yet I can't find any directory that is actually taking all this
>> >> space.
>> >> >  I've carefully looked through every single directory within the jail
>> >> trying
>> >> > to find something that's taking all that space, but to no avail.
>> >> >
>> >> > ----
>> >> > My system stats:
>> >> > # uname -a
>> >> > FreeBSD grim 8.1-RELEASE FreeBSD 8.1-RELEASE #0: Mon Jul 19  
>> 02:36:49 UTC
>> >> > 2010     root at mason.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  amd64
>> >> > # zpool get version tpool
>> >> > NAME   PROPERTY  VALUE    SOURCE
>> >> > tpool  version   14       default
>> >> > # zpool status
>> >> >   pool: tpool
>> >> >  state: ONLINE
>> >> >  scrub: none requested
>> >> > config:
>> >> >
>> >> >         NAME        STATE     READ WRITE CKSUM
>> >> >         tpool       ONLINE       0     0     0
>> >> >           mirror    ONLINE       0     0     0
>> >> >             ad4s1d  ONLINE       0     0     0
>> >> >             ad6s1d  ONLINE       0     0     0
>> >> >
>> >> > errors: No known data errors
>> >> >
>> >> > [ Note that I've also done a scrub recently ]
>> >>
>> >> 1) Have you checked using fstat to ensure that no file descriptors
>> >> remain open on any of your ZFS filesystems (not pools)?
>> >>
>> >> 2) Are you using compression on any of your ZFS filesystems?
>
> Andriy and Pawel,
>
> Do either of you have ideas as to what could cause the issue Torbjorn's
> experiencing?  I swear I remember some bug or quirk that got fixed with
> regards to free space on ZFS, but as has been proven time and time again
> my memory is horrible.  His kernel's 8.1-RELEASE dated July 19th.

IIRC the commit you talk about was by Martin (CCed). I do not know if  
it is (already) MFCed.

I'm not sure the bug you talk about is related to what Torbjorn is  
talking about. The fact that the free space is going down while the  
jail is shutdown (and I assume jls does not show his JID anymore, so  
all of its processes are really gone) points more to some other  
process (outside of the jail) which is filling some (maybe already  
deleted, so not visible anymore with du) file.

Bye,
Alexander.

-- 
A wide-eyed, innocent UNICORN, poised delicately in a MEADOW filled
with LILACS, LOLLIPOPS & small CHILDREN at the HUSH of twilight??

http://www.Leidinger.net    Alexander @ Leidinger.net: PGP ID = B0063FE7
http://www.FreeBSD.org       netchild @ FreeBSD.org  : PGP ID = 72077137