rrdtool performance tuning (fwd)
Richard A Steenbergen
ras at e-gerbil.net
Mon Nov 12 14:50:11 PST 2007
> Dear colleagues,
>
> any hints to tune rrdtool with ~30k rrd files (approx 2k target
> devices)?
>
> machine is mostly IO-bound, showing 100% disk load with 8 or sometimes
> even 3 mB/s, 300-400 tps (it's 2 SATA300 disks in gmirror)
Long and short of it, rrd sucks. I'm not sure whats sucks worse though,
the architecture, the code, the data format, the fact that you have to
exec open seek seek seek seek close for every one of those rrd files, or
the 10000 character long command line you get to throw at it to generate
complex graphs. :)
I was working on a complete rewrite of the rrd update code a couple months
back, but got distracted by other bigger projects and haven't had time to
finish it up. If anyone is interested, there is some alpha code (aka do
not run this on any .rrd files you value) at:
http://sourceforge.net/projects/librrd/
The goal was to implement an efficient C API which wouldn't open/close the
file with every cycle, use entirely mmap() (in stock code even when you
say to use mmap, there is still a ton of legacy code which seeks around
unnecessarily), and implement fine grained read/write locking for
concurrency. Personally though, I suspect the vast majority of the speed
improvements would come from just general code improvement rather than any
specific technique (e.g. removing 5 levels of indirection which are
completely unnecessary to accomplish a step, and which only exist because
the code is a running hack based on small contributions that "make it
work" from 100 different people who primarily write perl, all without any
overall design).
Pretty sure the remaining bug is in the CDP (consolidated data points)
code which creates missing data points in the event you've waited longer
than a PDP interval between updates. But more to the point, everything in
rrd_update_cdp() is something I haven't yet reverse engineered to figure
out what the actual goal is. If you can figure this out, you can probably
rewrite that entire block in 1/5th the code, like everywhere else. :)
Please let me know if there is any interest in this, I'll be happy to help
provide info on what I've done so far, until I find more free time. :)
--
Richard A Steenbergen <ras at e-gerbil.net> http://www.e-gerbil.net/ras
GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
More information about the freebsd-stable
mailing list