rm -rf stalls server

[Mehmet Erol Sanliturk]

On Thu, Mar 26, 2015 at 10:46 AM, Artem Kuchin <artem at artem.ru> wrote:

> Hello!
>
> I have a server with 100+ sites running nginx, apache,mysql
> The disk system is 2 TOSHIBA 3TB disk in geom mirror
>
> FS is UFS  SU+J
>
> tunefs -p /
> tunefs: POSIX.1e ACLs: (-a)                                disabled
> tunefs: NFSv4 ACLs: (-N)                                   disabled
> tunefs: MAC multilabel: (-l)                               disabled
> tunefs: soft updates: (-n)                                 enabled
> tunefs: soft update journaling: (-j)                       enabled
> tunefs: gjournal: (-J)                                     disabled
> tunefs: trim: (-t)                                         disabled
> tunefs: maximum blocks per file in a cylinder group: (-e)  4096
> tunefs: average file size: (-f)                            16384
> tunefs: average number of files in a directory: (-s)       64
> tunefs: minimum percentage of free space: (-m)             8%
> tunefs: space to hold for metadata blocks: (-k)            6408
> tunefs: optimization preference: (-o)                      time
> tunefs: volume label: (-L)                                 root
>
>
> Ram is 32GB
>
> load averages:  1.83,  1.77,  1.83
> usually about 500 processes are started
> Everything run nice and smooth.
>
> Periodically i need to run fs intensive tasks, like tar for backup.
> That kills server after 5 minutes of running. I worked it around using pv
> limiting bandwidth to 10M.
>
> But, sometimes i need to delete huge amount of small files in different
> dirs.
> Yes, i just need to do it.
> So, i do
> rm -rf  tree_root
>
> After i do it in 5 minutes process count goes over 1000, mysql is flooded
> with
> request it cannot complete. I can barely type anything.  Top shown tons of
> processes
> in 'ufs' status.
> So, I ctrl-C  rm command
> and kill -9 apache, otherwise it will take like 20 minutes to resume
> normal operations.
>
> Then i do
> /usr/bin/nice -n 20 rm -rf tree_root
>
> And it seems like rm -rf now is playing nice :)
>
> But, if load for web sites rises even for about 30% during this command
> server goes
> to stall again, 1000 process, hundereds of stuck mysql requests, tons of
> processes
> in ufs state.
>
> As i understand, freebsd does not have ionice
> So, any idea how to make this rm -rf work reaaaaaaly slow, not creating any
> load on hdd or metadata locks (i think it is actually fs meta data locks
> fault).
>
> Artem
>
>
>
>
>
>
>
>
> _______________________________________________
>
>



I am using  a script to delete .BAK files from many directories with  lines
like

rm path/*.BAK
find path -name '*.BAK'  -exec rm -f '{}' +


for each different path names which contains such files .


You may use such a script by inserting a delay causing statement between rm
- find pairs .

With these delays , it will not load CPU with a continuous successive
executions .

I do not know what a delay causing statement will be .

Perhaps the above hint may be useful for you .


Instead of using a single rm statement for all of the directories from
starting from the root directory , you may use different subdirectories in
a script like defined above .

In that way it will not be necessary to scan subdirectories which do not
contain files to remove , if they exist .



Another idea may the following ( I did not such an application . My opinion
is only theoretical which may not be true  ) .

Use a cron job definition having many script execution lines where each
script contains a small number of ( rm - find ) pairs and executed with
time interval spanning a defined time duration , for example one hour .

For example , with one minutes intervals such 60 scripts may be executed to
cover your subdirectories . Number of scripts and time intervals my be
planned with respect to your work loads and starting to execution of
scripts may be adjusted for least work load times of your server , for
example between hours 02:00 to 03:00 AM .



I think , it will not load your CPU continuously with a single execution
for all of the directories .





Thank you very much .


Mehmet Erol Sanliturk