inode state
Robert Watson
rwatson at freebsd.org
Tue Dec 9 10:38:40 PST 2003
On Tue, 9 Dec 2003, Fabian Thylmann wrote:
> I have a heavily used threaded server program running on one of my Dell
> Poweredge 1750 servers. Its a statistical analysis package for websites.
> Currently it analyses over 60 million requests a day, which (because of
> many different reasons) causes it to handle around 120 million http
> requests a day. At peaks around 1500 requests a second.
>
> The system stores most many statistics in memory which is flushed to
> disk in circles by a worker thread.
>
> Another big part is stored in an on-disk database which is mmap()'d into
> memory. Because we do not have enough memory to keep everything in
> memory at one time the mmap() system of course pages data in and out.
>
> When I look in systat -v I see that dirtybuf climbs to about 1700 and
> then they get flushed to disk, causing high disk usage of around 300-400
> tps whcih renders the disks useless for anything else.
>
> When those flushes occure, my apps state as displayed by top(1) gets
> into inode state, PRI is set to -14 and cpu usage rapidly drops. The
> program and ALL of its threads are stalled at that time. Those inode
> states take around 2 oe 3 seconds and happen every 30 seconds or so.
>
> In those 3 seconds we lose around 1500 hits at peak times for processing
> because the app can not handle them fast enough. This results in around
> 2 million or so hits lost over the day for processing.
>
> I am now wondering if anyone can explain to me why ALL threads and not
> just the threads that actually do I/O work get blocked when dirty
> buffers are flushed and what to do to fix this problem.
>
> I would be very happy if someone could reply and point me into the right
> direction!
You don't mention which version of FreeBSD you're running -- if 4.x, you
probably want to relink your application against the "linuxthreads" port.
This is because libc_r implements threads inside a single process without
the support of the kernel, which means that if the process is blocked in
kernel, all threads will be blocked in kernel. The linuxthreads package
uses a model similar to Linux's threading implementation (hence the name)
to allow the threads to be scheduled using lightweight versions of
processes (shared file descriptors, etc). This isn't quite
POSIX-compliant, but it works quite well for disk-bound applications such
as databases.
If you're running on 5.x, especially recent 5.1 or 5.2 prereleases, you
probably want to give libkse a try. It's the new m:n threading
implementation that will become the default in 5.3, and also permits
parallelism (only in a more POSIX-compliant way, and in theory offering
much greater scalability for large numbers of threads). I stick the
following lines in my /etc/libmap.conf on 5.x boxes to force all
applications linked against libc_r to use libkse instead:
libc_r.so.5 libkse.so.1
libc_r.so libkse.so
One particularly nice thing about the m:n thread support is that you can
run-time plug the thread library between several options (libc_r, libthr,
libkse) to pick the one that performs best for your application. Another
benefit of running with a non-libc_r threads package is that if you have
an SMP box, you'll see real parallelism.
Robert N M Watson FreeBSD Core Team, TrustedBSD Projects
robert at fledge.watson.org Senior Research Scientist, McAfee Research
More information about the freebsd-questions
mailing list