FreeBSD 9.0 hangs on heavy I/O
fjwcash at gmail.com
Tue May 29 19:40:48 UTC 2012
On Tue, May 29, 2012 at 12:34 PM, Freddie Cash <fjwcash at gmail.com> wrote:
> On Tue, May 29, 2012 at 12:26 PM, Kees Jan Koster <kjkoster at gmail.com> wrote:
>> I seem to have a problem where really heavy disk I/O is drowning my machine. I see hangs in the shell where I am logged on using ssh. Network connections get dropped for no apparent reason and some HTTP requests are served really slowly. Profiling the app code shows that the hangs are in completely random places. Operations that are no more than a few lines of code apart suddenly take seconds to complete.
>> In my search I seem to find that my machine is quite slow on the disk. I find that rather odd, given that the device in question is an SSD drive and it is a good bit faster than the WD drive that used to carry the data set that is accessed heavily. This drive is doing 1.5 times the throughput, but the hangs have not gone away.
>> To clarify, the data set used to live on ada2 (see the devlist below) which is a spinning disk. When I experienced intermittent hangs I plugged in an SSD drive (ada3 on the devlist) and moved the data there. This improved the MB's per second that are being written (it is mostly-write data) but has not changed the hangs. If anything, they got worse since.
>> Using gstat I notice that I/O service time is quite high. From the gstat below you can see that it takes just over 2s to servr the requests. The L(q) seems to never drop far below 100 and %busy hovers around 100% all day long. Can someone please help me troubleshoot that further? What can I do to make the underlying problem visible?
>> I should mention all data is referenced through cross-mountpoint symlinks, would that make a difference? Should I use canonical paths in the code instead?
>> All file systems are mounted "noatime, soft-updates".
> You may want to play around with gshed, the GEOM Scheduler.
> Matt Dillon did a bunch of tests comparing FreeBSD+UFS to
> DragonflyBSD+HAMMER and found that FreeBSD starves read threads in
> order to satisfy write threads (or the other way around?). But,
> adding gsched into the mix helped things immensely, allowing mixed
> reads/writes to better shares disk I/O resources.
> I'll see if I can dig up a link to his testing e-mail messages.
Here's the post, part of a thread on benchmarking RAID controllers:
fjwcash at gmail.com
More information about the freebsd-stable