Poor read() performance, and I can't profile it

[Kirk Strauser]

On Thursday 05 June 2008, Kirk Strauser wrote:

> I was testing the same software on my desktop PC when I noticed that it
> ran *much* faster, and found that it was spending only about 1% as much
> time in the kernel on Linux as it was on FreeBSD.

I'm almost ready to give up on this.  I've gone as far as completely rewriting the 
original C++ program into straightforward C, and still the performance is terrible on 
FreeBSD versus Linux.  

On Linux:

$ time ./cdbf /tmp/invoice.dbf >/dev/null
./cdbf /tmp/invoice.dbf > /dev/null  42.65s user 20.09s system 71% cpu 1:28.15 total

On FreeBSD:



Also note that on the FreeBSD machine, I have enough RAM that to buffer the entire 
file, and in practice gstat shows that the drives are idle for subsequent runs after 
the first one.

Right now my code looks a lot like:

   for(recordnum = 0; recordnum < recordcount; recordnum++) {
	buf = malloc(recordlength);
	fread(buf, recordlength, 1, dbffile);

        /* Do stuff with buf */

        memoblock = getmemoblock(buf);
        /* Skip to the requested block if we're not already there */
	if(memoblock != currentmemofileblock) {
	    currentmemofileblock = memoblock;
	    fseek(memofile, currentmemofileblock * memoblocksize, SEEK_SET);
	}
	memohead = malloc(memoblocksize);
	fread(memohead, memoblocksize, 1, memofile);
	currentmemofileblock++;

        /* Do stuff with memohead */

        free(memohead);
	free(buf);
    }

...where recordlength == 13 in this one case.  Given that the whole file is buffered in 
RAM, the small reads shouldn't make a difference, should they?  I've played with 
setvbuf() and it shaves off a few percent of runtime, but nothing to write home about.

Now, memofile gets quite a lot of seeks.  Again, that shouldn't make too much of a 
difference if it's already buffered in RAM, should it?  setvbuf() on that file that 
gets lots of random access actually made performance worse.

What else can I do to make my code run as well on FreeBSD as it does on a much wimpier 
Linux machine?  I'm almost to the point of throwing in the towel and making a Linux 
server to do nothing more than run this one program if I can't FreeBSD's performance 
more on parity, and I honestly never thought I'd be considering that.

I'll gladly give shell access with my code and sample data files if anyone is 
interested in testing it.
-- 
Kirk Strauser