ZFS Problem - full disk, can't recover space :(.

[Jeremy Chadwick]

On Sun, Mar 27, 2011 at 11:01:04AM +0100, Dr Josef Karthauser wrote:
> On 27 Mar 2011, at 10:41, Jeremy Chadwick wrote:
> > I'm curious about something -- we use RELENG_8 systems with a mirror
> > zpool (kinda funny how I did it too, since the system only has 2 disks)
> > for /home.  Our SpamAssassin configuration is set to obviously writes to
> > $user/.spamassassin/bayes_* files.  Yet, we do not see this sparse file
> > problem that others are reporting.
> > 
> > $ df -k /home
> > Filesystem 1024-blocks      Used     Avail Capacity  Mounted on
> > data/home    239144704 107238740 131905963    45%    /home
> > $ zfs list data/home
> > NAME        USED  AVAIL  REFER  MOUNTPOINT
> > data/home   102G   126G   102G  /home
> > 
> > $ zpool status data
> >  pool: data
> > state: ONLINE
> > scrub: resilver completed after 0h9m with 0 errors on Wed Oct 20 03:08:22 2010
> > config:
> > 
> >        NAME         STATE     READ WRITE CKSUM
> >        data         ONLINE       0     0     0
> >          mirror     ONLINE       0     0     0
> >            ada1     ONLINE       0     0     0
> >            ada0s1g  ONLINE       0     0     0  26.0G resilvered
> > 
> > $ grep bayes /usr/local/etc/mail/spamassassin/local.cf
> > use_bayes 1
> > bayes_auto_learn 1
> > bayes_ignore_header X-Bogosity
> > bayes_ignore_header X-Spam-Flag
> > bayes_ignore_header X-Spam-Status
> > 
> > $ ls -l .spamassassin/
> > total 4085
> > -rw-------    1 jdc       users      102192 Mar 27 02:30 bayes_journal
> > -rw-------    1 jdc       users      360448 Mar 27 02:30 bayes_seen
> > -rw-------    1 jdc       users     4947968 Mar 27 02:30 bayes_toks
> > -rw-------    1 jdc       users        8719 Mar 20 04:11 user_prefs
> 
> No idea what caused it, but whenever I ran the bayes expiry it created a new file that just blew up and filled all the available space. I've got around the issue temporarily. I used 'swapoff' to recover a 4Gb swap partition, created a UFS and mounted that in the jail in question. After rsyncing the bayes database to that disk I was able to run an expire with no trouble at all, so it wasn't that the bayes was corrupt or anything. I've now copied it back and it runs fine. I expect that the problem will reoccur at some inconvenient point in the future.

Not to say you're wrong -- there are lots of people who experience this
problem it seems -- but I can't reproduce it.

$ cd .spamassassin/
$ ls -l bayes_*
-rw-------    1 jdc       users       42888 Mar 27 03:13 bayes_journal
-rw-------    1 jdc       users      360448 Mar 27 03:13 bayes_seen
-rw-------    1 jdc       users     4947968 Mar 27 03:13 bayes_toks
$ rm bayes_*
$ mail jdc
> Subject: testing bayes
> sfddsfdfs
> i like snakes
> .
> EOT
$ ls -l bayes_*
-rw-------    1 jdc       users     131072 Mar 27 03:38 bayes_seen
-rw-------    1 jdc       users     131072 Mar 27 03:38 bayes_toks

The system in question:

amd64
FreeBSD 8.1-STABLE #0: Wed Oct 20 00:54:42 PDT 2010

This system is currently running ZFS pool version 15.
All pools are formatted using this version.

This system is currently running ZFS filesystem version 4.
All filesystems are formatted with the current version.

Dunno what to say other than that.  :-(

-- 
| Jeremy Chadwick                                   jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.               PGP 4BD6C0CB |