ggate + gmirror write performance woes

[Nikolay Pavlov]

On Thursday,  5 April 2007 at 16:15:35 -0400, Sven Willenberger wrote:
> On Thu, 2007-04-05 at 17:38 +0100, Tom Judge wrote:
> > Dmitriy Kirhlarov wrote:
> > > On Thu, Apr 05, 2007 at 10:58:56AM -0400, Sven Willenberger wrote:
> > >> I am trying to set up a HA type system involving two identical boxes and
> > >> have gone through the following to set up the systems:
> > >>
> > >> Slave server: 
> > >> ggated -R 196608 -S 196608
> > >> (exporting /dev/amrd1 )
> > >> net.inet.tcp.sendspace: 65536
> > >> net.inet.tcp.recvspace: 131072
> > > 
> > > Try
> > > net.local.stream.recvspace=65535
> > > net.local.stream.sendspace=65535
> > > 
> > > Also, try increase this sysctls with
> > > net.inet.tcp.rfc1323=1
> > > 
> > > I use it on FreeBSD 5.x with:
> > > net.inet.tcp.sendspace=131072
> > > net.inet.tcp.recvspace=131072
> > > net.local.stream.recvspace=65535
> > > net.local.stream.sendspace=65535
> > > 
> > > ggated -R 1048576 -S 1048576
> > > ggatec -R 1048576 -S 1048576
> > > 
> > > WBR.
> > > Dmitriy
> > > _______________________________________________
> > > freebsd-stable at freebsd.org mailing list
> > > http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> > > To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"
> > 
> > 
> > I have seen sustained writes of 30Mb/s using the following configuration:
> > 
> > cat /boot/loader.conf
> > kern.ipc.nmbclusters="32768"
> > 
> > cat /etc/sysctl.conf
> > net.inet.tcp.sendspace=1048576
> > net.inet.tcp.recvspace=1048576
> > 
> > Server:
> > /sbin/ggated -S 1310720 -R 1310720 -a 172.31.0.18 /etc/gg.exports
> > 
> > Client:
> > /sbin/ggatec create -q 2048 -t 5 -S 1310720 -R 1310720 172.31.0.18 
> > /dev/amrd0s2
> > 
> > The raid array is a RAID 1 volume on a dell PERC4 (Dell PE1850) with 
> > adaptive read ahead and write back caching.
> > 
> > Tom
> 
> I have tried both the settings ideas suggested above but I cannot even
> get out of the gate with those. Setting net.inet.tcp.{send,recv}space to
> anything higher that 131072 results in ggated bailing with the error:
> # ggated -v -a 10.10.0.19
> info: Reading exports file (/etc/gg.exports).
> debug: Added 10.10.0.0/24 /dev/amrd1 RW to exports list.
> debug: Added 10.10.0.0/24 /dev/amrd3 RW to exports list.
> info: Exporting 2 object(s).
> error: Cannot open stream socket: No buffer space available.
> error: Exiting.

For values of net.inet.tcp.{send,recv}space more than
524288 you also need to adjust kern.ipc.maxsockbuf

Try this configuration for example:
kern.ipc.maxsockbuf=2049152
net.inet.tcp.recvspace=1024576
net.inet.tcp.sendspace=1024576

> 
> setting net.inet.tcp.{send,recv}space to 131072 allows me to start
> ggated with the default R and S values of 131072; anything higher
> results in "no buffer space" errors. At 131072 ggated starts but then I
> cannot even open a new connection (like ssh) to the server as the ssh
> client bails with "no buffer space available".
> 
> more information:
> # netstat -m
> 514/641/1155 mbufs in use (current/cache/total)
> 512/284/796/32768 mbuf clusters in use (current/cache/total/max)
> 512/256 mbuf+clusters out of packet secondary zone in use
> (current/cache)
> 0/0/0/0 4k (page size) jumbo clusters in use (current/cache/total/max)
> 0/0/0/0 9k jumbo clusters in use (current/cache/total/max)
> 0/0/0/0 16k jumbo clusters in use (current/cache/total/max)
> 1152K/728K/1880K bytes allocated to network (current/cache/total)
> 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
> 0/0/0 requests for jumbo clusters denied (4k/9k/16k)
> 0/4/6656 sfbufs in use (current/peak/max)
> 0 requests for sfbufs denied
> 0 requests for sfbufs delayed
> 0 requests for I/O initiated by sendfile
> 0 calls to protocol drain routines
> 
> This is on a FreeBSD 6.2-RELENG box i386 SMP using the amr driver (SATA
> Raid using LSiMegaRaid.
> 
> The odd thing is that even after I set the send and recvspace down to
> values like 65536, I continue to get the no buffer error when trying to
> connect to it remotely again.
> 
> Sven
> 
> _______________________________________________
> freebsd-stable at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"

-- 
======================================================================  
- Best regards, Nikolay Pavlov. <<<-----------------------------------    
======================================================================