rrdtool performance tuning (fwd)

Richard A Steenbergen ras at e-gerbil.net
Mon Nov 12 14:50:11 PST 2007


> Dear colleagues,
> 
> any hints to tune rrdtool with ~30k rrd files (approx 2k target 
> devices)?
> 
> machine is mostly IO-bound, showing 100% disk load with 8 or sometimes 
> even 3 mB/s, 300-400 tps (it's 2 SATA300 disks in gmirror)

Long and short of it, rrd sucks. I'm not sure whats sucks worse though, 
the architecture, the code, the data format, the fact that you have to 
exec open seek seek seek seek close for every one of those rrd files, or 
the 10000 character long command line you get to throw at it to generate 
complex graphs. :)

I was working on a complete rewrite of the rrd update code a couple months 
back, but got distracted by other bigger projects and haven't had time to 
finish it up. If anyone is interested, there is some alpha code (aka do 
not run this on any .rrd files you value) at:

http://sourceforge.net/projects/librrd/

The goal was to implement an efficient C API which wouldn't open/close the 
file with every cycle, use entirely mmap() (in stock code even when you 
say to use mmap, there is still a ton of legacy code which seeks around 
unnecessarily), and implement fine grained read/write locking for 
concurrency. Personally though, I suspect the vast majority of the speed 
improvements would come from just general code improvement rather than any 
specific technique (e.g. removing 5 levels of indirection which are 
completely unnecessary to accomplish a step, and which only exist because 
the code is a running hack based on small contributions that "make it 
work" from 100 different people who primarily write perl, all without any 
overall design).

Pretty sure the remaining bug is in the CDP (consolidated data points) 
code which creates missing data points in the event you've waited longer 
than a PDP interval between updates. But more to the point, everything in 
rrd_update_cdp() is something I haven't yet reverse engineered to figure 
out what the actual goal is. If you can figure this out, you can probably 
rewrite that entire block in 1/5th the code, like everywhere else. :) 
Please let me know if there is any interest in this, I'll be happy to help 
provide info on what I've done so far, until I find more free time. :)

-- 
Richard A Steenbergen <ras at e-gerbil.net>       http://www.e-gerbil.net/ras
GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)


More information about the freebsd-stable mailing list