ggate + gmirror write performance woes

[Sven Willenberger]

On Thu, 2007-04-05 at 17:38 +0100, Tom Judge wrote:
> Dmitriy Kirhlarov wrote:
> > On Thu, Apr 05, 2007 at 10:58:56AM -0400, Sven Willenberger wrote:
> >> I am trying to set up a HA type system involving two identical boxes and
> >> have gone through the following to set up the systems:
> >>
> >> Slave server: 
> >> ggated -R 196608 -S 196608
> >> (exporting /dev/amrd1 )
> >> net.inet.tcp.sendspace: 65536
> >> net.inet.tcp.recvspace: 131072
> > 
> > Try
> > net.local.stream.recvspace=65535
> > net.local.stream.sendspace=65535
> > 
> > Also, try increase this sysctls with
> > net.inet.tcp.rfc1323=1
> > 
> > I use it on FreeBSD 5.x with:
> > net.inet.tcp.sendspace=131072
> > net.inet.tcp.recvspace=131072
> > net.local.stream.recvspace=65535
> > net.local.stream.sendspace=65535
> > 
> > ggated -R 1048576 -S 1048576
> > ggatec -R 1048576 -S 1048576
> > 
> > WBR.
> > Dmitriy
> > _______________________________________________
> > freebsd-stable at freebsd.org mailing list
> > http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> > To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"
> 
> 
> I have seen sustained writes of 30Mb/s using the following configuration:
> 
> cat /boot/loader.conf
> kern.ipc.nmbclusters="32768"
> 
> cat /etc/sysctl.conf
> net.inet.tcp.sendspace=1048576
> net.inet.tcp.recvspace=1048576
> 
> Server:
> /sbin/ggated -S 1310720 -R 1310720 -a 172.31.0.18 /etc/gg.exports
> 
> Client:
> /sbin/ggatec create -q 2048 -t 5 -S 1310720 -R 1310720 172.31.0.18 
> /dev/amrd0s2
> 
> The raid array is a RAID 1 volume on a dell PERC4 (Dell PE1850) with 
> adaptive read ahead and write back caching.
> 
> Tom

I have tried both the settings ideas suggested above but I cannot even
get out of the gate with those. Setting net.inet.tcp.{send,recv}space to
anything higher that 131072 results in ggated bailing with the error:
# ggated -v -a 10.10.0.19
info: Reading exports file (/etc/gg.exports).
debug: Added 10.10.0.0/24 /dev/amrd1 RW to exports list.
debug: Added 10.10.0.0/24 /dev/amrd3 RW to exports list.
info: Exporting 2 object(s).
error: Cannot open stream socket: No buffer space available.
error: Exiting.

setting net.inet.tcp.{send,recv}space to 131072 allows me to start
ggated with the default R and S values of 131072; anything higher
results in "no buffer space" errors. At 131072 ggated starts but then I
cannot even open a new connection (like ssh) to the server as the ssh
client bails with "no buffer space available".

more information:
# netstat -m
514/641/1155 mbufs in use (current/cache/total)
512/284/796/32768 mbuf clusters in use (current/cache/total/max)
512/256 mbuf+clusters out of packet secondary zone in use
(current/cache)
0/0/0/0 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/0 9k jumbo clusters in use (current/cache/total/max)
0/0/0/0 16k jumbo clusters in use (current/cache/total/max)
1152K/728K/1880K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0/4/6656 sfbufs in use (current/peak/max)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile
0 calls to protocol drain routines

This is on a FreeBSD 6.2-RELENG box i386 SMP using the amr driver (SATA
Raid using LSiMegaRaid.

The odd thing is that even after I set the send and recvspace down to
values like 65536, I continue to get the no buffer error when trying to
connect to it remotely again.

Sven