amrd disk performance drop after running under high load
lol at chistydom.ru
Sun Nov 11 09:26:52 PST 2007
Kris Kennaway wrote:
>>> In the "good" case you are getting a much higher interrupt rate but
>>> with the data you provided I can't tell where from. You need to run
>>> vmstat -i at regular intervals (e.g. every 10 seconds for a minute)
>>> during the "good" and "bad" times, since it only provides counters
>>> and an average rate over the uptime of the system.
>> Now I'm running 10-process lighttpd and the problem became no so big.
>> I collected interrupt stats and it shows no relation beetween
>> ionterrupts and slowdowns. Here is it:
>> Also I have similiar statistics on mutex profiling and it shows
>> there's no problem in mutexes.
>> I have no idea what else to check.
> I don't know what this graph is showing me :) When precisely is the
> system behaving poorly?
Take a look at "Disk Load %" picture at
At ~ 17:00, 03:00-04:00, 13:00-14:00, 00:30-01:30, 11:00-13:00 it shows
peaks of disk activity which really never happen. As I said in the
beginning of the thread in this "peak" moments disk becomes slow and
vmstat shows 100% disk load while performing < 10 tps. Other grafs at
this page shows that there's no relation to interrupts rate of amr or em
device. You advised me to check it.
When I was using single-process lighttpd the problem was much harder as
you can see at http://220.127.116.11/gprof/graph/ . At first picture on
this page you can see disk load peaks at 18:00 and 15:00 which leaded to
decreasing network output because disk was too slow.
Back in this thread we suspected UMA mutexes. In order to check it I
collected mutex profiling stats and draw graphs over time and they also
didn't show anything interesting. All mutex graphs were smooth while
disk load peaks. http://18.104.22.168/gprof/mtx-graph/mtxgifnew/
With best regards,
More information about the freebsd-hackers