inode state

[Fabian Thylmann]

Hi,

I have a heavily used threaded server program running on one of my Dell Poweredge 1750 servers. Its a statistical analysis package for websites. Currently it analyses over 60 million requests a day, which (because of many different reasons) causes it to handle around 120 million http requests a day. At peaks around 1500 requests a second.

The system stores most many statistics in memory which is flushed to disk in circles by a worker thread.

Another big part is stored in an on-disk database which is mmap()'d into memory. Because we do not have enough memory to keep everything in memory at one time the mmap() system of course pages data in and out.

When I look in systat -v I see that dirtybuf climbs to about 1700 and then they get flushed to disk, causing high disk usage of around 300-400 tps whcih renders the disks useless for anything else.

When those flushes occure, my apps state as displayed by top(1) gets into inode state, PRI is set to -14 and cpu usage rapidly drops. The program and ALL of its threads are stalled at that time. Those inode states take around 2 oe 3 seconds and happen every 30 seconds or so.

In those 3 seconds we lose around 1500 hits at peak times for processing because the app can not handle them fast enough. This results in around 2 million or so hits lost over the day for processing.

I am now wondering if anyone can explain to me why ALL threads and not just the threads that actually do I/O work get blocked when dirty buffers are flushed and what to do to fix this problem.

I would be very happy if someone could reply and point me into the right direction!

Fabian Thylmann