rm -rf stalls server

[Artem Kuchin]

Hello!

I have a server with 100+ sites running nginx, apache,mysql
The disk system is 2 TOSHIBA 3TB disk in geom mirror

FS is UFS  SU+J

tunefs -p /
tunefs: POSIX.1e ACLs: (-a)                                disabled
tunefs: NFSv4 ACLs: (-N)                                   disabled
tunefs: MAC multilabel: (-l)                               disabled
tunefs: soft updates: (-n)                                 enabled
tunefs: soft update journaling: (-j)                       enabled
tunefs: gjournal: (-J)                                     disabled
tunefs: trim: (-t)                                         disabled
tunefs: maximum blocks per file in a cylinder group: (-e)  4096
tunefs: average file size: (-f)                            16384
tunefs: average number of files in a directory: (-s)       64
tunefs: minimum percentage of free space: (-m)             8%
tunefs: space to hold for metadata blocks: (-k)            6408
tunefs: optimization preference: (-o)                      time
tunefs: volume label: (-L)                                 root


Ram is 32GB

load averages:  1.83,  1.77,  1.83
usually about 500 processes are started
Everything run nice and smooth.

Periodically i need to run fs intensive tasks, like tar for backup.
That kills server after 5 minutes of running. I worked it around using pv
limiting bandwidth to 10M.

But, sometimes i need to delete huge amount of small files in different 
dirs.
Yes, i just need to do it.
So, i do
rm -rf  tree_root

After i do it in 5 minutes process count goes over 1000, mysql is 
flooded with
request it cannot complete. I can barely type anything.  Top shown tons 
of processes
in 'ufs' status.
So, I ctrl-C  rm command
and kill -9 apache, otherwise it will take like 20 minutes to resume 
normal operations.

Then i do
/usr/bin/nice -n 20 rm -rf tree_root

And it seems like rm -rf now is playing nice :)

But, if load for web sites rises even for about 30% during this command 
server goes
to stall again, 1000 process, hundereds of stuck mysql requests, tons of 
processes
in ufs state.

As i understand, freebsd does not have ionice
So, any idea how to make this rm -rf work reaaaaaaly slow, not creating any
load on hdd or metadata locks (i think it is actually fs meta data locks 
fault).

Artem