amrd disk performance drop after running under high load

Alexey Popov lol at
Sun Nov 11 10:52:23 PST 2007


Kris Kennaway wrote:
>>> In the "good" case you are getting a much higher interrupt rate but 
>>> with the data you provided I can't tell where from.  You need to run 
>>> vmstat -i at regular intervals (e.g. every 10 seconds for a minute) 
>>> during the "good" and "bad" times, since it only provides counters 
>>> and an average rate over the uptime of the system.
>> Now I'm running 10-process lighttpd and the problem became no so big.
>> I collected interrupt stats and it shows no relation beetween 
>> ionterrupts and slowdowns. Here is it:
>> Also I have similiar statistics on mutex profiling and it shows 
>> there's no problem in mutexes. 
>> I have no idea what else to check.

> I don't know what this graph is showing me :)  When precisely is the 
> system behaving poorly?
Take a look at "Disk Load %" picture at

At ~ 17:00, 03:00-04:00, 13:00-14:00, 00:30-01:30, 11:00-13:00 it shows 
peaks of disk activity which really never happen. As I said in the 
beginning of the thread in this "peak" moments disk becomes slow and 
vmstat shows 100% disk load while performing < 10 tps. Other grafs at 
this page shows that there's no relation to interrupts rate of amr or em 
device. You advised me to check it.

When I was using single-process lighttpd the problem was much harder as 
you can see at . At first picture on 
this page you can see disk load peaks at 18:00 and 15:00 which leaded to 
decreasing network output because disk was too slow.

Back in this thread we suspected UMA mutexes. In order to check it I 
collected mutex profiling stats and draw graphs over time and they also 
didn't show anything interesting. All mutex graphs were smooth while 
disk load peaks.

With best regards,
Alexey Popov
freebsd-hackers at mailing list
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe at"

More information about the freebsd-stable mailing list