Slow disk write speeds over network

Eric Anderson anderson at
Tue Jun 10 08:27:33 PDT 2003

Good news, but not done yet.. Keep reading:

Terry Lambert wrote:
[..snippity snip..]
> Swap cables with another box.
> BTW: 4 Gigabit cards in one box, with you going to local disk...
> you've got about 8 cards worth of traffic over your PCI bus.

I'm going to a RAID50 (hardware), and I know there's the PCI bus limits 
- I'm not planning on filling 4 Gig E's at once continually..

> Unless this is a PCI-X based box, you are most likely livelocked;
> even if it's a PCI-X based box, you could still be livelocked.
> You haven't said if you were using UDP or TCP for the mounts;
> you should definitely use TCP with FreeBSD NFS servers; it's
> also just generally a good idea, since UDP frags act as a fixed
> non-sliding window: NFS over UDP sucks.

Most clients are TCP, but some are still UDP (due to bugs in unmentioned 
linux distros nfs clients).

> Also, you haven't said whether you are using aliases on your
> network cards; aliases and NFS tend to interact badly.

Nope, no aliases.. I have one card on each network, with one IP per 
card.  I have full subnets (/24) full of P4's trying to slam the NFS 
server for data all the time..

> Finally, you probably want to tweak some sysctl's, e.g.
> 	net.inet.ip.check_interface=0
> 	net.inet.tcp.inflight_enable=1
> 	net.inet.tcp.inflight_debug=0
> 	net.inet.tcp.msl=3000
> 	net.inet.tcp.inflight_min=6100
> 	net.isr.enable=1

Ok - done.. some where defaults, and I couldn't find net.isr.enable.. 
Did I need to config something on my kernel for it to show up?
Also, can you explain any of those tweaks?

> Given your overloading of your bus, that last one is probably
> the most important one: it enables direct dispatch.
> You'll also want to enable DEVICE_POLLING in your kernel
> config file (assuming you have a good ethernet card whose
> driver supports it):
> 	options HZ=2000

Well, the LINT file says only a few cards support it - not sure if I 
should trust that or not, but I have Intel PRO/1000T Server Adapters - 
which should be good enough cards to support it.. I've also put 100Mbit 
cards in place of the gige's for now to make sure I wasn't hitting a 
GigE problem or negotiation problem..

> ...and yet more sysctl's for this:
> 	kern.polling.enable=1
> 	kern.polling.user_frac=50	# 0..100; whatever works best
> If you've got a really terrible Gigabit Ethernet card, then
> you may be copying all your packets over again (e.g. m_pullup()),
> and that could be eating your bus, too.

Ok, so the end result is that after playing around with sysctl's, I've 
found that the tcp transfers are doing 20MB/s over FTP, but my NFS is 
around 1-2MB/s - still slow.. So we've cleared up some tcp issues, but 
yet still NFS is stinky..

Any more ideas?


Eric Anderson	   Systems Administrator      Centaur Technology
Attitudes are contagious, is yours worth catching?

More information about the freebsd-performance mailing list