What causes random disk access slow down

[Jin Guojun]

A 6-7 years old Xeon dual 2.4MHz CPU machine runs FreeBSD 6.4-Release 
suddenly becomes
slow on some tasks requiring disk access. Typical things like ls, 
objdump etc. 
Be more specific, a couple of minutes objdump became a several hours job.
A several seconds "ls -RC" became a 15-minute task (see output below).

It sounds like a hard drive problem, but run sequential disk test on all 
drives, their throughput
meet the  original disk spec and disks run very quite, at random disk 
access, disks generate
 some rigid noise, so it looks like a random disk access problem.
This machine has two IDE PATA drives (ignore da0 -- a USB stick), but No 
error message has
been recorded in dmesg for any dirve a couple of weeks after the problem 
happened.

Machine has been rebooted a few times after slowness occurred, but it 
won't help.
Is there anyway/any tool to find out what is going wrong in the system?

-Jin

[165] bsd-ms: ls -RC > Dir
3.756u 19.402s 15:29.37 2.4%    30+2938k 49120+76io 0pf+0w

monitored from the other terms --
[138] bsd-ms: ll ~/Dir
-rw-r--r--  1 src  wheel  6152192 Oct 27 14:53 /home/users/src/Dir
[139] bsd-ms: ll ~/Dir
-rw-r--r--  1 src  wheel  8019968 Oct 27 14:56 /home/users/src/Dir
[140] bsd-ms: ll ~/Dir
-rw-r--r--  1 src  wheel  9915957 Oct 27 14:58 /home/users/src/Dir

      tty             ad0              ad1              da0             cpu
 tin tout  KB/t tps  MB/s   KB/t tps  MB/s   KB/t tps  MB/s  us ni sy in id
   9  365  9.14   6  0.05  12.08   6  0.07  121.91   0  0.00   2  0  1  0 97
   0 1020 11.75  87  0.99  17.82   7  0.13   0.00   0  0.00  76  0 14  0 10
   0 1005  8.54 262  2.19  52.94  23  1.21   0.00   0  0.00  61  0 29  1  9
   0  893  7.54 184  1.36  85.76  34  2.82   0.00   0  0.00  53  0 32  1 14
   0  551  3.35 265  0.87   9.38   4  0.04   0.00   0  0.00  47  0 33  1 19
   0  594  6.81 201  1.33  37.82   4  0.14   0.00   0  0.00  54  0 16  0 30
   0 1106  3.54 252  0.87  55.19  17  0.93   0.00   0  0.00  39  0 33  1 27
   0  393  2.88 223  0.63  11.43   2  0.03   0.00   0  0.00  67  0 31  1  1
   0  644  4.81 165  0.77  16.00   0  0.01   0.00   0  0.00  87  0 12  1  1
  27  339 10.39 180  1.82  15.18  11  0.17   0.00   0  0.00  86  0 13  0  0
  32  130  5.06 146  0.72  23.40  46  1.04   0.00   0  0.00  86  0  8  1  5
  32  267  8.39 138  1.13  61.09   4  0.22   0.00   0  0.00  73  0 26  1  0
  33  340  8.75 222  1.90  61.54   4  0.26   0.00   0  0.00  78  0 21  1  0
  32  595  5.85 154  0.88  12.20   3  0.04   0.00   0  0.00  87  0 12  1  0
  32  288  5.28 147  0.76   6.00   1  0.01   0.00   0  0.00  86  0 13  1  0