gmirror + ggated question

[kurtseel]

Patrick Tracanelli wrote:
> kurtseel escreveu:
>>
>> I built a mirror of a local drive and a ggated backed device. I ran 
>> iozone on it
>> and it runs along fine until a certain point when it slows down to a 
>> near stand
>> still. It doesn't break the mirror or crash the system, but it does 
>> slow the system
>> down to a near stop.
>> I kill the iozone, and a short time later I can login and then :
>>
>> # df
>> Filesystem            1K-blocks    Used    Avail Capacity  Mounted on
>> /dev/mirror/thinkcs1a   1012974  155780   776158    17%    /
>> devfs                         1       1        0   100%    /dev
>> /dev/mirror/thinkcs1e  85469448 1163474 77468420     1%    /usr
>> /dev/mirror/thinkcs1d   4058062   40426  3692992     1%    /var
>> [root@ ~/temp]# gmirror status
>>         Name    Status  Components
>> mirror/thinkc  COMPLETE  ad0
>>                         ggate0
>>
>> And all seems normal again. Seems like it has to do with big files ...
>> This is the same configuration I used in :
>> http://bsdtips.utcorp.net/mediawiki/index.php/Mirroring_over_network
>> This is where the iozone gets stuck :
>
> Did you try raising send and receive buffers on ggated? I found myself 
> confortable with -S and -R around 512k-780k. I didnt, however, did an 
> iozone stress test, just a production test (real load) before going 
> production.
>
> Try raising the buffer and let us know about your tests. TCP_NODELAY 
> is also worth trying.
>
 Makes sense. So now I get this :

Test (/root/benchmarks) > ggated -v -R 262144 -S 262144 /etc/ggated.conf
info: Reading exports file (/etc/ggated.conf).
debug: Added 10.200.1.200/32 /dev/ad10 RW to exports list.
info: Exporting 1 object(s).
error: Cannot open stream socket: No buffer space available.
error: Exiting.

Test (/root/benchmarks) > ggated -v -R 524288 -S 524288 /etc/ggated.conf
info: Reading exports file (/etc/ggated.conf).
debug: Added 10.200.1.200/32 /dev/ad10 RW to exports list.
info: Exporting 1 object(s).
error: Cannot open stream socket: No buffer space available.
error: Exiting.

 I have raised

sysctl net.inet.tcp.sendspace=4194304
sysctl net.inet.tcp.recvspace=4194304
sysctl kern.ipc.maxsockbuf=2097152

 Which I saw in a posting ...

 It even happens here :

Test (/root/benchmarks) > ggated -v -R 1 -S 1 /etc/ggated.conf
info: Reading exports file (/etc/ggated.conf).
debug: Added 10.200.1.200/32 /dev/ad10 RW to exports list.
info: Exporting 1 object(s).
error: Cannot open stream socket: No buffer space available.
error: Exiting.