Strange ZFS problem, filesystem claims to be full when clearly
not full
Torbjorn Kristoffersen
torbjoern at gmail.com
Wed Sep 29 22:11:12 UTC 2010
I'm at a complete loss here. I shut down the jail completely, and I am
watching the jail's ZFS filesystem grow as we speak. No process is using
it. It only grows in "df" and "zfs list", I can't find any files that are
growing. I have to re-set the quota to be higher and higher to accommodate
the space.
On Wed, Sep 29, 2010 at 10:46 PM, Torbjorn Kristoffersen <
torbjoern at gmail.com> wrote:
> Hi Jeremy.
>
> 1) I checked now, and found nothing extraordinary. Just processes that have
> been running for a long while, such as screen, cron, sshd, bash, irssi,
> syslogd, etc.
>
> 2) No compression used on this zfs filesystem (or any of the others).
>
> I completedly stopped the jail now, and removed some of the directories
> with the most data in them, but to no avail.
>
>
> On Wed, Sep 29, 2010 at 9:25 PM, Jeremy Chadwick <freebsd at jdc.parodius.com
> > wrote:
>
>> On Wed, Sep 29, 2010 at 08:46:38PM +0200, Torbjorn Kristoffersen wrote:
>> > I have a ZFS "tank" called tpool, the server runs a couple of jails
>> (each
>> > with a zfs filesystem). There is a problem with one of these
>> filesystems.
>> > First, its disk usage as shown in ``df -h'':
>> > ...
>> > tpool/rb.org 100G 95G 4.6G 95% /jails/rb.org
>> > ...
>> >
>> > The command ``zfs list'' shows the same:
>> > ..
>> > tpool/rb.org 95.4G 4.56G 95.4G /jails/rb.org
>> > ..
>> >
>> > However, there is a very mysterious problem somewhere.
>> > Something inside this jail is eating diskspace, but we can't find any
>> > directories that is actually taking the diskspace. We first suspected
>> either
>> > fetchmail or spamassassin of causing a lot of space to be used, since
>> some
>> > of their directories were huge. (These were later deleted, and which is
>> why
>> > you see that 4.6GB is now available, before that 0GB was available).
>> >
>> > However, we can't find *any trace* of an actual directory or file that
>> is
>> > taking all the spac.e
>> >
>> > Take this for instance:
>> >
>> > outsidejail# du -sh rb.org
>> > 43G rb.org
>> >
>> > How can this be? df and zfs are showing that the entire drive is nearly
>> > full, yet I can't find any directory that is actually taking all this
>> space.
>> > I've carefully looked through every single directory within the jail
>> trying
>> > to find something that's taking all that space, but to no avail.
>> >
>> > ----
>> > My system stats:
>> > # uname -a
>> > FreeBSD grim 8.1-RELEASE FreeBSD 8.1-RELEASE #0: Mon Jul 19 02:36:49 UTC
>> > 2010 root at mason.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64
>> > # zpool get version tpool
>> > NAME PROPERTY VALUE SOURCE
>> > tpool version 14 default
>> > # zpool status
>> > pool: tpool
>> > state: ONLINE
>> > scrub: none requested
>> > config:
>> >
>> > NAME STATE READ WRITE CKSUM
>> > tpool ONLINE 0 0 0
>> > mirror ONLINE 0 0 0
>> > ad4s1d ONLINE 0 0 0
>> > ad6s1d ONLINE 0 0 0
>> >
>> > errors: No known data errors
>> >
>> > [ Note that I've also done a scrub recently ]
>>
>> 1) Have you checked using fstat to ensure that no file descriptors
>> remain open on any of your ZFS filesystems (not pools)?
>>
>> 2) Are you using compression on any of your ZFS filesystems?
>>
>> --
>> | Jeremy Chadwick jdc at parodius.com |
>> | Parodius Networking http://www.parodius.com/ |
>> | UNIX Systems Administrator Mountain View, CA, USA |
>> | Making life hard for others since 1977. PGP: 4BD6C0CB |
>>
>>
>
More information about the freebsd-fs
mailing list