zfs send/recv invalid data

[Steven Hartland]

----- Original Message ----- 
From: "Larry Rosenman" <ler at lerctr.org>
To: "Steven Hartland" <killing at multiplay.co.uk>
Cc: "Martin Simmons" <martin at lispworks.com>; <tevans.uk at googlemail.com>; <freebsd-fs at freebsd.org>; <ronald-freebsd8 at klop.yi.org>
Sent: Thursday, March 07, 2013 1:14 AM
Subject: Re: zfs send/recv invalid data


> On 2013-03-06 17:48, Steven Hartland wrote:
>> ----- Original Message ----- From: "Larry Rosenman" <ler at lerctr.org>
>> To: "Steven Hartland" <killing at multiplay.co.uk>
>> Cc: "Martin Simmons" <martin at lispworks.com>;
>> <tevans.uk at googlemail.com>; <freebsd-fs at freebsd.org>;
>> <ronald-freebsd8 at klop.yi.org>
>> Sent: Wednesday, March 06, 2013 11:16 PM
>> Subject: Re: zfs send/recv invalid data
>> 
>> 
>>> On 2013-03-06 17:00, Steven Hartland wrote:
>>>> ----- Original Message ----- From: "Larry Rosenman" <ler at lerctr.org>
>>>>> # cat send.stream | ssh root at home openssl md5
>>>>> (stdin)= 9cd1d73ea8411f1c222bc90e7bea3d33
>>>>> # uname -a
>>>>> FreeBSD thebighonker.lerctr.org 8.3-STABLE FreeBSD 8.3-STABLE #54 
>>>>> r247820: Mon Mar  4 18:08:11 CST 2013     
>>>>> root at thebighonker.lerctr.org:/usr/obj/usr/src/sys/THEBIGHONKER  
>>>>> amd64
>>>>> #
>>>>> # exec csh
>>>>> root at borg:/home/ler # pwd
>>>>> /home/ler
>>>>> root at borg:/home/ler # ssh tbh "cat 
>>>>> /home/ler/public_html/ZFS_RECV/send.stream | ssh 
>>>>> root at home.lerctr.org openssl md5"
>>>>> (stdin)= 9cd1d73ea8411f1c222bc90e7bea3d33
>>>>> root at borg:/home/ler # uname -a
>>>>> FreeBSD borg.lerctr.org 10.0-CURRENT FreeBSD 10.0-CURRENT #124 
>>>>> r247826: Mon Mar  4 19:59:08 CST 2013     
>>>>> root at borg.lerctr.org:/usr/obj/usr/src/sys/BORG-DTRACE  amd64
>>>>> root at borg:/home/ler #
>>>> When you run a command from ssh it will run as the target users 
>>>> shell
>>>> so for csh if there is something bad in .cshrc or iirc .profile then
>>>> you may get something output purely by logging in.
>>>> So ensure you shell config's for the user on the target machine are
>>>> clean.
>>>> You can confirm this by comparing the output from the following
>>>> two commands run from the sending machine:-
>>>> md5 send.stream
>>>> cat send.stream | ssh root at home.lerctr.org "cat" |md5
>>>> If these two commands produce different md5's then the login
>>>> shell for root on home.lerctr.org is outputing something
>>>> even when there is no terminal.
>>>> You should be able to see what the shell is outputting using:-
>>>> echo -n '' | ssh root at home.lerctr.org "cat" > 1.log
>>>> After running this 1.log should be 0 bytes if its not your
>>>> shell is definitely your problem and I'd start by looking
>>>> at .cshrc and .profile for this user, or even the globals
>>>> for these in /etc/
>>>> 
>>> I've done that, and get identical md5's.....
>>> I'm at a loss.
>> 
>> I believe what you ran was similar but not identical, if you
>> check the quoting subtly different.
>> 
>> I'm not sure that will have any effect but both of the above
>> are worth running just to confirm.
>> 
>>    Regards
>>    Steve
> $ ^DConnection to thebighonker.lerctr.org closed.
> $ ssh root at tbh
> Last login: Tue Mar  5 06:51:46 2013 from cpe-72-182-19-1
> FreeBSD 8.3-STABLE (THEBIGHONKER) #54 r247820: Mon Mar  4 18:08:11 CST 
> 2013
> 
> Welcome to FreeBSD!
> 
> root at thebighonker:~ # pwd
> /root
> root at thebighonker:~ # md5 /home/ler/public_html/ZFS_RECV/send
> send-file-snap-list  send.stream          send.uname
> root at thebighonker:~ # md5 /home/ler/public_html/ZFS_RECV/send.stream
> MD5 (/home/ler/public_html/ZFS_RECV/send.stream) = 
> 9cd1d73ea8411f1c222bc90e7bea3d33
> root at thebighonker:~ # cat /home/ler/public_html/ZFS_RECV/send.stream 
> |ssh root at home.lerctr.org "cat"|md5
> 9cd1d73ea8411f1c222bc90e7bea3d33
> root at thebighonker:~ #
> 
> 
> looks like it's clean.....

So to summarise, just to make sure I've got everything clear so far:-
1. sending the stream over ssh from the source -> target results in
the same md5 (from the test above)
2. sending the stream over ssh from source -> target piped into
zfs recv fails
3. scp'ing the stream file from the source to the target and then 
importing it via zfs recv works

Yes / No?

If so this is very strange indeed.

I'd like to confirm that adding the zfs recv to the picture isn't changing
the dynamic such that the stdin results are changing and thus eliminating
ssh as the source of any corruption to do this can you test:-
md5 send.stream
cat send.stream | ssh root at home.lerctr.org "tee tea.stream | zfs recv <flags>"
ssh root at home.lerctr.org "md5 tea.stream"

    Regards
    Steve

================================================
This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. 

In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337
or return the E.mail to postmaster at multiplay.co.uk.