Sendfile performance regression from 4.x to 5.x

[Kevin Day]

We've got two nearly identically configured servers, serving the exact 
same load. Quick config:

Dual P4 2.8Ghz with HTT enabled, 2GB RAM
ahc SCSI adapter, with 4 drives in a vinum RAID5 config
Dual bge gigabit network cards

Server 1 is running 4.8
Server 2 is running 5.2.1

Other than OS version, everything is identical for all practical 
purposes.
I'm running thttpd with USE_SENDFILE enabled. Both servers are 
receiving approximately equal numbers of requests to download some 
large files. (5MB to 200MB)

Server 1 can easily push 300-500mbps. I'm pretty sure it could go 
higher, we just run into bandwidth limits at that point. With the caps 
removed, we've hit 800mbps for a short while.
Server 2 seems to cap out around 80mbps, with thttpd spending most of 
it's time in "Giant" or "kqueue" according to top, the CPU around 90% 
idle.

What's really strange is that right when thttpd starts up on 5.2, it 
will start sending 200mbps or higher for the first 10-15 minutes, and 
then start tapering off back down to 80mbps. It doesn't look like a 
thttpd bug, since it seems to work flawlessly on 4.x.

I know a lot of work is being done on locking, sendfile and kqueue fun, 
so I'm not complaining. I just wanted someone to know this was 
happening, and hopefully have someone point out something obvious I'm 
missing that could be causing this... Anyone run into this before?

-- Kevin