filesystem: 12h to delete 32GB of data (4 million files)

[Olivier Mueller]

Thanks for your answer Bill!  (and to Will as well),

Some more infos I gathered a few minutes ago: 

[~/templates_c]$ date; du -s -m ; date
Wed May  6 13:35:15 CEST 2009
  2652      .
Wed May  6 13:52:36 CEST 2009

[~/templates_c]$ date ; find . | wc -l ; date
Wed May  6 13:52:56 CEST 2009
  305461
Wed May  6 14:09:39 CEST 2009


So this is on the system after a complete cache cleanup (at 00h00).
300'000 files and 2.6GB.  So this night, there were probably around 3-4
million files to delete.

Deletion may take time, but 20 minutes juste to _count_ all the files
seems pretty long to me...   I think I'll say a word to the developers
to let them tune their caching system a bit :)



On Wed, 2009-05-06 at 08:48 -0400, Bill Moran wrote:
> With lots of small files, the time involved is far less dependent on
> the size of data, and much more dependent on the number of files, and
> the resultant number of directory entries that need to be updated.
> "Lots" isn't a particularly accurate count of the # of files, but if
> you're talking web cache files, I'll guess they average 5k each, which
> means you had 6 million files.  df -i would have been more useful in
> the output above.

Thanks, noted for next time.  Now it looks like that:
Filesystem  1M-blocks  Used Avail Capacity iused    ifree %iused  Mounted on
/dev/da0s1f    128631 70544 47795    60% 1913875 15114219   11%   /usr


> This brings a number of questions up:
> * Are you _sure_ softupdates is enabled on that partition?  That's
>   going to make the biggest improvement in speed.

According to "mount" output, yes.  I found no specific message about
that in the syslog or dmesg.

> * Are these 7200RPM disks or 15,000?  Again, going to make a big
>   difference.

HP 146GB 6G SAS 10K SFF DP ENT HDD  (15k were not available at the time
the servers were ordered)
( http://h18004.www1.hp.com/products/servers/proliantstorage/serial/sas/index.html )

> * If apache was still running, is it possible that it was creating
>   enough disk activity to slow the activity down?  Running
>   top -m io will show you how much disk IO each process is creating.

Yes, apache was still running, but the activity was quite low (it was
during the night, and the webpage doesn't get so many hits before 9 am
local time)

While watching "top -m io", the "du" or "find" takes between 80 and 99%,
so I guess it's not the probleme here:

  PID    UID          VCSW  IVCSW   READ  WRITE  FAULT  TOTAL PERCENT COMMAND
87996   1002            59     56      0      0      0      0   0.00% php
45389   1002            35     25      0      0      2      2   0.84% php
 3964   1002             0      0      0      0      0      0   0.00% httpd
 3822   1002           151     98      0      0      0      0   0.00% httpd
 3005   1002             0      0      0      0      0      0   0.00% httpd
 4129   1002             0      0      0      0      0      0   0.00% httpd
 3971   1002             0      0      0      0      0      0   0.00% httpd
 4231   1002             1      0      0      0      0      0   0.00% httpd
 4132      0           234      5    234      0      0    234  97.91% find
98862   1002             1      0      0      0      0      0   0.00% top
  609      0             0      0      0      0      0      0   0.00% snmpd
[...]


> * When you compared the speed to your laptop, did you delete 6 million
>   files from the laptop?  If you deleted a single 30G file, then you're
>   comparing apples to atom bombs.

Yes sorry, I know :)

> If this is a directory that you blow away on a regular schedule, you'd
> do much better to make it a dedicated partition and simply reformat
> it.

Yes, it is one of the best options. My initial goal was to delete all
files older than N days by cron  (find | xargs | rm, etc.), but if each
cronjob takes 2 hours (and takes so much cpu time), it's probably not
the best way.  

I'll make some more tests on an test-server later this week and speak
with the devs. Thanks again for your very constructive feedback! 

Regards,
Olivier