Multi-machine mirroring choices

Sven W sven at dmv.com
Mon Jul 21 15:23:59 UTC 2008



Pete French presumably uttered the following on 07/21/08 07:08:
>> The *big* issue I have right now is dealing with the slave machine going
>> down. Once the master no longer has a connection to the ggated devices,
>> all processes trying to use the device hang in D status. I have tried
>> pkill'ing ggatec to no avail and ggatec destroy returns a message of
>> gctl being busy. Trying to ggatec destroy -f panics the machine.
> 
> Oddly enough, this was the issue I had with iscsi which made me move
> to using ggated instead. On our machines I use '-t 10' as an argument to
> ggatec, and this makes it timeout once the connection has been down for
> a certain amount of time. I am using gmirror on top, not ZFS, and this
> handled the drive vanishing from the mirror quite happily. I haven't
> tried it with ZFS, which may not like having the device suddenly dissapear.
> 
> -pete.

What I have found is that the master machine will lock up if the slave disappears 
during a large file transfer. I tested this by setting up zpool mirror on the master 
using a ggatec device from the slave. Then I:

pkill'ed ggated on the slave machine.

dd if=/dev/zero of=/data1/testfile2 bs=16k count=8192   [128MB] on the master

The dd command finished and the /var/log/messages showed I/O errors to the slave 
drive as expected. Messages also showed ggatec trying to reconnect every 10 seconds 
(ggatec was started with the -t 10 parameter).

Finally zfs marked the drive unavailable which then allowed me to ggatec destroy -u 
0 without getting the "ioctl(/dev/ggctl): Device busy" error message. (By the way, 
using ggatec destroy does not kill the "ggatec create" that created the process to 
begin with, I had to pkill ggatec to get that stop - bug?)

The above behavior would be acceptable for multi-machine mirroring as it would be 
scriptable.

The problem comes with Large writes. I tried to repeat the above with

dd if=/dev/zero of=/data1/testfile2 bs=16k count=32768 [512MB]

which then locks zfs,  and ultimately the system itself. It seems once the write 
size/buffer is full, zfs is unable to fail/unavail the slave drive and the entire 
system becomes unresponsive (cannot even ssh into it).

The bottom line is that without some type of "timeout" or "time to fail" (bad I/O to 
fail?) zpool + ggate[cd] seems to be an unworkable solution. This is actually a 
shame as the recover process swapping from master to slave and back again was so 
much cleaner and faster than using gmirror.


Sven


More information about the freebsd-stable mailing list