Consistent inodes between distinct machines

[Bernd Walter]

On Fri, May 02, 2008 at 03:40:11PM -0500, Eric Anderson wrote:
> On Apr 17, 2008, at 3:43 AM, Attila Nagy wrote:
> 
> >Hello,
> >
> >I have several NFS servers, where the service must be available  
> >0-24. The servers are mounted read only on the clients and I've  
> >solved the problem of maintaining consistent inodes between them by  
> >rsyncing an UFS image and mounting it via md on the NFS servers.
> >The machines have a common IP address with CARP, so if one of them  
> >falls out, the other(s) can take over.
> >
> >This works nice, but rsyncing multi gigabyte files are becoming more  
> >and more annoying, so I've wondered whether it would be possible to  
> >get constant inodes between machines via alternative ways.
> 
> 
> Why not avoid syncing multi-gigabyte files by splitting your huge FS  
> image into many smaller say 512MB files, then use md and geom concat/ 
> stripe/etc to make them all one image that you mount?

Where would be the positive effect by doing this?
FFS distributes data over the media, so all the small files changes
in almost every case and you have to checksum-compare the whole virtual
disk anyway.
With multiple files the syncing is more complex. For example a normal
rsync run can garantie that you get a complete file synced or none
at all, but this doesn't work out of the box with multiple files, so
you risk half updated data.

Nevertheless I think that the UFS/NFS combo is not very good for this
problem.

With ZFS send/receive however inode numbers are consistent.
Together with the differential stream creation it is quite efficient
to sync large volumes as well.
[75]cicely14# zfs send data/arm-elf at 2008-05-03 | zfs receive -v data/test
receiving full stream of data/arm-elf at 2008-05-03 into data/test at 2008-05-03
received 126Mb stream in 28 seconds (4.50Mb/sec)
0.008u 5.046s 0:27.93 18.0%     53+2246k 0+0io 0pf+0w
[56]cicely14# ls -ali /usr/local/arm-elf/bin/
total 22585
147 drwxr-xr-x   2 root  wheel       20 Mar 25  2006 .
  3 drwxr-xr-x  11 root  wheel       11 Dec 25 04:58 ..
154 -rwxr-xr-x   1 root  wheel  1514107 Mar 25  2006 arm-elf-addr2line
150 -rwxr-xr-x   2 root  wheel  1495219 Mar 25  2006 arm-elf-ar
159 -rwxr-xr-x   2 root  wheel  2275463 Mar 25  2006 arm-elf-as
158 -rwxr-xr-x   1 root  wheel  1481234 Mar 25  2006 arm-elf-c++filt
163 -rwxr-xr-x   1 root  wheel   300233 Mar 25  2006 arm-elf-cpp
164 -rwxr-xr-x   2 root  wheel   296938 Mar 25  2006 arm-elf-gcc
164 -rwxr-xr-x   2 root  wheel   296938 Mar 25  2006 arm-elf-gcc-4.1.0
162 -rwxr-xr-x   1 root  wheel    15949 Mar 25  2006 arm-elf-gccbug
161 -rwxr-xr-x   1 root  wheel   126715 Mar 25  2006 arm-elf-gcov
160 -rwxr-xr-x   2 root  wheel  2162285 Mar 25  2006 arm-elf-ld
156 -rwxr-xr-x   2 root  wheel  1541809 Mar 25  2006 arm-elf-nm
153 -rwxr-xr-x   1 root  wheel  1871104 Mar 25  2006 arm-elf-objcopy
149 -rwxr-xr-x   2 root  wheel  2008424 Mar 25  2006 arm-elf-objdump
152 -rwxr-xr-x   2 root  wheel  1495214 Mar 25  2006 arm-elf-ranlib
155 -rwxr-xr-x   1 root  wheel   389000 Mar 25  2006 arm-elf-readelf
148 -rwxr-xr-x   1 root  wheel  1430608 Mar 25  2006 arm-elf-size
151 -rwxr-xr-x   1 root  wheel  1412788 Mar 25  2006 arm-elf-strings
157 -rwxr-xr-x   2 root  wheel  1871103 Mar 25  2006 arm-elf-strip
[57]cicely14# ls -ali /data/test/bin/
total 22585
147 drwxr-xr-x   2 root  wheel       20 Mar 25  2006 .
  3 drwxr-xr-x  11 root  wheel       11 Dec 25 04:58 ..
154 -rwxr-xr-x   1 root  wheel  1514107 Mar 25  2006 arm-elf-addr2line
150 -rwxr-xr-x   2 root  wheel  1495219 Mar 25  2006 arm-elf-ar
159 -rwxr-xr-x   2 root  wheel  2275463 Mar 25  2006 arm-elf-as
158 -rwxr-xr-x   1 root  wheel  1481234 Mar 25  2006 arm-elf-c++filt
163 -rwxr-xr-x   1 root  wheel   300233 Mar 25  2006 arm-elf-cpp
164 -rwxr-xr-x   2 root  wheel   296938 Mar 25  2006 arm-elf-gcc
164 -rwxr-xr-x   2 root  wheel   296938 Mar 25  2006 arm-elf-gcc-4.1.0
162 -rwxr-xr-x   1 root  wheel    15949 Mar 25  2006 arm-elf-gccbug
161 -rwxr-xr-x   1 root  wheel   126715 Mar 25  2006 arm-elf-gcov
160 -rwxr-xr-x   2 root  wheel  2162285 Mar 25  2006 arm-elf-ld
156 -rwxr-xr-x   2 root  wheel  1541809 Mar 25  2006 arm-elf-nm
153 -rwxr-xr-x   1 root  wheel  1871104 Mar 25  2006 arm-elf-objcopy
149 -rwxr-xr-x   2 root  wheel  2008424 Mar 25  2006 arm-elf-objdump
152 -rwxr-xr-x   2 root  wheel  1495214 Mar 25  2006 arm-elf-ranlib
155 -rwxr-xr-x   1 root  wheel   389000 Mar 25  2006 arm-elf-readelf
148 -rwxr-xr-x   1 root  wheel  1430608 Mar 25  2006 arm-elf-size
151 -rwxr-xr-x   1 root  wheel  1412788 Mar 25  2006 arm-elf-strings
157 -rwxr-xr-x   2 root  wheel  1871103 Mar 25  2006 arm-elf-strip

-- 
B.Walter <bernd at bwct.de> http://www.bwct.de
Modbus/TCP Ethernet I/O Baugruppen, ARM basierte FreeBSD Rechner uvm.