filesystem: 12h to delete 32GB of data

[cpghost]

On Wed, May 06, 2009 at 05:34:24PM +0200, Wojciech Puchar wrote:
> > -> it took about 12 hours to delete these 30GB of files and
> > 
> > sub-directories (smarty cache files: many small files in many dirs).
> > It's a little bit surprising, as it's on a recent HP proliant DL360 g5
> > with SAS disks (Raid1) running freebsd 6.x
> > ( /dev/da0s1f on /usr (ufs, local, soft-updates) )
> 
> if you would use no raid or software raid it will behave normally.
> 
> it takes <30 minutes for me to delete 300GB of squid files on 
> ordinary SATA disk , millions of small files.

Alternatively, you could assign a dedicated filesystem for the cache
and when cleaning up:
  * stop the app (or disable caching),
  * umount
  * newfs
  * mount
  * restart the app (or reenable caching).

newfs is MUCH faster than manually deleting gazillions of files.  If
you don't like the (small) downtime during newfs, you could also play
with two or more dedicated filesystems, and rotate between them (though
that would be a waste of disk space).

I can't recall how many times I've used a fresh newfs-ed filesystem
instead of removing stuff one file at a time.

-cpghost.

-- 
Cordula's Web. http://www.cordula.ws/