ZFS trouble: unbelievably large files created

[jhell]

On 10/15/2010 18:44, David Samms wrote:
> On 10/15/10 15:09, Oliver Fromme wrote:
>>   >  ls -kls shawxp/var/amavis/.spamBAD
>>   >  total 13352131
>>   >        771 -rw-------  1 110  110           684032 Oct 15 12:02
>>   >  auto-whitelist
>>   >          1 -rw-------  1 110  110               40 Oct 15 12:37
>> bayes.lock
>>   >       1284 -rw-------  1 110  110          1294336 Oct 15 12:38
>> bayes_seen
>>   >       4229 -rw-------  1 110  110          4227072 Oct 15 12:38
>> bayes_toks
>>   >    5025103 -rw-------  1 110  110     553184002048 Oct 15 12:38
>>   >  bayes_toks.expire3515
>>   >    8320745 -rw-------  1 110  110  140743122878464 Oct 15 12:14
>>   >  bayes_toks.expire97254
>>
>> Ok, so those files are so-called "sparse" files, i.e. they
>> contain holes that don't actually occupy disk space.
>>
>> The numbers in the first column indicate the amount of
>> physical disk space allocated (in KB).  That's about 5 GB
>> for the first file and 8 GB for the second (this is also
>> consistent with the "total" value in the first line of the
>> ls output, i.e. about 13 GB).
>>
>> That's still quite big, but certainly not in the TB range.
>> I do not know why amavis creates such large sparse files,
>> though.
> 
> Under UFS I don't observe amavis creating the sparse files.  The problem
> is fairly repeatable, not on demand, but occurs several times a day.
> High disk activity is a sure sign amavis is creating large files.  Task
> shuts down normaly and cpu load is low.
> 
> This is a production server but I have moved all critical customer jails
> back to UFS so can do testing if anyone has any ideas of what to look for.
> 

I do not believe this to be a problem with amavis/amavisd, lets get that
out of the way.

Why I say this is because I have experienced the same problem on a
server with UFS and p5-Mail-Spamassassin.

I was logged into that server doing some work and there was plenty of
disk space to do the work as it consisted of copying a ton of space
eating files to the file-system where this happened. After operations
were done, cleaned up the disk properly restoring space back for normal
usage and then logged out.

I came back the next day only to find my INBOX full of error messages
and other junk from that server saying things like can't write to disk
and such error messages.

What it turned out to be was the same files that you had mentioned above
but in my ~/.spamassassin/.bayes*. One of the files just like yours had
grown to eat up the entire amount of disk space that was left besides
the 8% that was reserved.

I tried to find the offending email messages that might have caused this
to happen but in some odd ~7 hours total time did not come up with a
conclusion and could not afford to waste further time on that issue.

IMHO this has to do with bayes and the usage of dbm, or db4* and
something in a messages header or body that triggers it to happen but do
not have enough time or conclusive evidence yet to suggest anything further.

Ultimately I removed the offending files and went about business as
usual until I see it happen again. Also tuned the quota for myself to be
a much lower amount disk space so I should get an email as soon as it
happens again without effecting the rest of operations on that disk.


Regards,

-- 

 jhell,v