gmirror + ggated question

[Patrick Tracanelli]

kurtseel escreveu:
> Patrick Tracanelli wrote:
>> kurtseel escreveu:
>>>
>>> I built a mirror of a local drive and a ggated backed device. I ran 
>>> iozone on it
>>> and it runs along fine until a certain point when it slows down to a 
>>> near stand
>>> still. It doesn't break the mirror or crash the system, but it does 
>>> slow the system
>>> down to a near stop.
>>> I kill the iozone, and a short time later I can login and then :
>>>
>>> # df
>>> Filesystem            1K-blocks    Used    Avail Capacity  Mounted on
>>> /dev/mirror/thinkcs1a   1012974  155780   776158    17%    /
>>> devfs                         1       1        0   100%    /dev
>>> /dev/mirror/thinkcs1e  85469448 1163474 77468420     1%    /usr
>>> /dev/mirror/thinkcs1d   4058062   40426  3692992     1%    /var
>>> [root@ ~/temp]# gmirror status
>>>         Name    Status  Components
>>> mirror/thinkc  COMPLETE  ad0
>>>                         ggate0
>>>
>>> And all seems normal again. Seems like it has to do with big files ...
>>> This is the same configuration I used in :
>>> http://bsdtips.utcorp.net/mediawiki/index.php/Mirroring_over_network
>>> This is where the iozone gets stuck :
>>
>> Did you try raising send and receive buffers on ggated? I found myself 
>> confortable with -S and -R around 512k-780k. I didnt, however, did an 
>> iozone stress test, just a production test (real load) before going 
>> production.
>>
>> Try raising the buffer and let us know about your tests. TCP_NODELAY 
>> is also worth trying.
>>
> Makes sense. So now I get this :
> 
> Test (/root/benchmarks) > ggated -v -R 262144 -S 262144 /etc/ggated.conf
> info: Reading exports file (/etc/ggated.conf).
> debug: Added 10.200.1.200/32 /dev/ad10 RW to exports list.
> info: Exporting 1 object(s).
> error: Cannot open stream socket: No buffer space available.
> error: Exiting.
> 
> Test (/root/benchmarks) > ggated -v -R 524288 -S 524288 /etc/ggated.conf
> info: Reading exports file (/etc/ggated.conf).
> debug: Added 10.200.1.200/32 /dev/ad10 RW to exports list.
> info: Exporting 1 object(s).
> error: Cannot open stream socket: No buffer space available.
> error: Exiting.
> 
> I have raised
> 
> sysctl net.inet.tcp.sendspace=4194304
> sysctl net.inet.tcp.recvspace=4194304
> sysctl kern.ipc.maxsockbuf=2097152
> 
> Which I saw in a posting ...
> 
> It even happens here :
> 
> Test (/root/benchmarks) > ggated -v -R 1 -S 1 /etc/ggated.conf
> info: Reading exports file (/etc/ggated.conf).
> debug: Added 10.200.1.200/32 /dev/ad10 RW to exports list.
> info: Exporting 1 object(s).
> error: Cannot open stream socket: No buffer space available.
> error: Exiting.
> 

Seems that you are out of buffer spance and it is not related to ggated, 
since -R 1 and -S 1 would not demand a bounch of extra memory. In any 
case, tuning kern.ipc.maxsockbuf should be enough.

If I raise to 512K I get out of buffer space too, on the default value. 
However, just raising it solves the prob:

(eksffa at claire)~# sysctl -qw kern.ipc.maxsockbuf=`echo "524288*2" | bc `
kern.ipc.maxsockbuf: 262144 -> 1048576

(eksffa at claire)~# ggated -R 524288 -S 524288 -v
info: Reading exports file (/etc/gg.exports).
debug: Added 10.0.0.0/24 /dev/ad12 RO to exports list.
info: Exporting 1 object(s).
info: Listen on port: 3080.

And so, I can import ggate0 on the other host.

Try figuring out with netstat -m why you ran out of buffer. Also, I 
believe it can be related to the fact you have raised recvspace and 
sndspace way too high. I dont think it makes any sense raising it over 
64k on 100Mbit network, or 128-512k on 1Gbit network. You have raised 
'em up to 4MB :) Lower down to the default 32k (send) / 64k (recv) 
first. If you are on 1Gbit or 10Gbit you can, later, tray raising on 
multiple of 32K untill the point you see it makes sense (where it makes 
positive difference on your benchs).

If it is anyhow relevant, I run it on 6.2-STABLE, cvsuped on Sept 24th, 
with the patches PJD mentioned applied.

-- 
Patrick Tracanelli

FreeBSD Brasil LTDA.
(31) 3281-9633 / 3281-3547
316601 at sip.freebsdbrasil.com.br
http://www.freebsdbrasil.com.br
"Long live Hanin Elias, Kim Deal!"