amrd disk performance drop after running under high load
Kris Kennaway
kris at FreeBSD.org
Wed Oct 17 01:20:42 PDT 2007
Alexey Popov wrote:
>>> This is very unlikely, because I have 5 another video storage servers
>>> of the same hardware and software configurations and they feel good.
>> Clearly something is different about them, though. If you can
>> characterize exactly what that is then it will help.
> I can't see any difference but a date of installation. Really I compared
> all parameters and got nothing interesting.
>
>>> At first glance one can say that problem is in Dell's x850 series or
>>> amr(4), but we run this hardware on many other projects and they work
>>> well. Also Linux on them works.
>>
>> OK but there is no evidence in what you posted so far that amr is
>> involved in any way. There is convincing evidence that it is the mbuf
>> issue.
> Why are you sure this is the mbuf issue?
Because that is the only problem shown in the data you posted.
> For example, if there is a real
> problem with amr or VM causing disk slowdown, then when it occurs the
> network subsystem will have another load pattern. Instead of just quick
> sending large amounts of data, the system will have to accept large
> amount of sumultaneous connections waiting for data. Can this cause high
> mbuf contention?
I'd expect to see evidence of the main problem.
>>> And few hours ago I received feed back from Andrzej Tobola, he has
>>> the same problem on FreeBSD 7 with Promise ATA software mirror:
>> Well, he didnt provide any evidence yet that it is the same problem,
>> so let's not become confused by feelings :)
> I think he is telling about 100% disk busy while processing ~5
> transfers/sec.
"% busy" as reported by gstat doesn't mean what you think it does. What
is the I/O response time? That's the meaningful statistic for
evaluating I/O load. Also you didnt post about this.
>>> So I can conclude that FreeBSD has a long standing bug in VM that
>>> could be triggered when serving large amount of static data (much
>>> bigger than memory size) on high rates. Possibly this only applies to
>>> large files like mp3 or video.
>> It is possible, we have further work to do to conclude this though.
> I forgot to mention I have pmc and kgmon profiling for good and bad
> times. But I have not enough knowledge to interpret it right and not
> sure if it can help.
pmc would be useful.
> Also now I run nginx instead of lighttpd on one of the problematic
> servers. It seems to work much better - sometimes there is a peaks in
> disk load, but disk does not become very slow and network output does
> not change. The difference of nginx is that it runs in multiple
> processes, while lighttpd by default has only one process. Now I
> configured lighttpd on other server to run in multiple workers. I'll see
> if it helps.
>
> What else can i try?
Still waiting on the vmstat -z output.
Kris
More information about the freebsd-hackers
mailing list