FreeBSD tunnels / performance et'al (gif/tun etc.)

[Robert Watson]

On Fri, 23 Jan 2004, Karl Pielorz wrote:

> > On Tue, 20 Jan 2004, Karl Pielorz wrote:
> >
> >> I've just setup a FreeBSD tunnel (we've tried both gif and tun [via
> >> nos-tun]) now between two fairly large networks of machines...
> >
> > What version of FreeBSD are you using?  If using FreeBSD 5.x, you may well
> > want to switch to 4.x for at least one more minor version, as interrupt
> > latency hasn't been optimized in 5.x yet since the move to interrupt
> > threads, and the network stack also runs with Giant in 5.2 out of the
> > box.  I wouldn't think this would hurt you as much as seen below, but
> > it's worth keeping in mind.
> >
> > Also, I would generally expect gif, gre, et al, to be faster than
> > tun-based tunneling, as they avoid the trip through userspace, which
> > involves a number of packet copies.
> 
> We're already using 4.9. I also take your point about gif being quicker
> than switching to user space and back (And, in testing - tun was indeed
> even slower than gif). 
> 
> In the end we fixed this problem by putting stupidly fast machines at
> each end (i.e. P4 2.6Ghz) - we also made some tweaks to the tcp sysctls
> (such as disabling delayed acks, and closing the window size down) -
> which also seemed to help. 
> 
> I'm just wondering if it was something 'weird' such as the delay over
> the tunnel being on average 'just the right delay time' to cause
> problems that you wouldn't get on a LAN or something? :) 

I agree that something sounds weird -- I've had no problem tunneling
hundreds of megabits using similar hardware to what you're using, and what
sounds like a similar configuration.  So it seems like someting is going
on.  Do you have any load information available on the systems -- i.e.,
interrupt rate as measured by vmstat, cpu usage, etc?  Are you using natd
or other address space translation?

Robert N M Watson             FreeBSD Core Team, TrustedBSD Projects
robert at fledge.watson.org      Senior Research Scientist, McAfee Research