disappointing rebuild speed using ggatec and ggated

[Ahnjoan Amous]

I'm attempting a RAID 1 mirror with ggate[c,d] between two hosts using
6.1-release and am disappointed by the rebuild rate.

I have tried using Ethernet cards supported by fxp and re drivers,
have read advice on how to tune to improve network performance.  I
have also attempted ggate[c,d] configuration using an md rather than
ad device but can't seem to figure out what might cause the problem
being encountered.

Searching the lists and web has lead make the following modifications.

*** These are quadruple what the defaults were on my system***
sysctl net.inet.tcp.sendspace=131072
sysctl net.inet.tcp.recvspace=262144
sysctl kern.ipc.maxsockbuf=1048576
sysctl kern.ipc.nmbclusters=35584

*** Kernel config ***
options         HZ=8000
options         DEVICE_POLLING

*** ifconfig***
enable POLLING on each machine with re0

With the following modifications, my rebuild has improved.  Instead of
stalling out at 2% (of a 146G mirror) the rebuild seems to steadily
continue albeit very slowly.  Today an entire manual rebuild (non auto
sync for gm device) took over 17 hours over a crossover cable between
two "re" interfaces.  I don't expect that the rebuild would happen at
wire speed but guess it might be possible to finish in less than 4
hours.

I do not have the gm device or ggate device mounted on the primary
host during the rebuild.  I am not connecting to either the primary or
failover host for any services. ( I hope this communicates that both
hosts are not taxed by external sources )

Using "systat –vm 1" shows that the machine comes close to 0%  idle
while the mirror is actually rebuilding.  Using "gstat" shows that the
ggate device on the primary comes close to 100% busy while the mirror
is actually rebuilding.

For some reason the rebuild seems to stop (systat –vm 1 goes to 80%
idle, and gstat goes to 0% busy for ggate) every once in a while.  I
can't pin point why it stops  rebuilding and hope that I have just
missed someone commenting that I should do something obvious to fix
this issue.

Thanks
Ahnjoan