zfs send/recv invalid data

[Larry Rosenman]

On 2013-03-07 13:08, Steven Hartland wrote:
> ----- Original Message ----- From: "Larry Rosenman" <ler at lerctr.org>
> To: "Steven Hartland" <killing at multiplay.co.uk>
> Cc: "Martin Simmons" <martin at lispworks.com>;
> <tevans.uk at googlemail.com>; <freebsd-fs at freebsd.org>;
> <ronald-freebsd8 at klop.yi.org>
> Sent: Thursday, March 07, 2013 1:14 AM
> Subject: Re: zfs send/recv invalid data
> 
> 
>> On 2013-03-06 17:48, Steven Hartland wrote:
>>> ----- Original Message ----- From: "Larry Rosenman" <ler at lerctr.org>
>>> To: "Steven Hartland" <killing at multiplay.co.uk>
>>> Cc: "Martin Simmons" <martin at lispworks.com>;
>>> <tevans.uk at googlemail.com>; <freebsd-fs at freebsd.org>;
>>> <ronald-freebsd8 at klop.yi.org>
>>> Sent: Wednesday, March 06, 2013 11:16 PM
>>> Subject: Re: zfs send/recv invalid data
>>> 
>>> 
>>>> On 2013-03-06 17:00, Steven Hartland wrote:
>>>>> ----- Original Message ----- From: "Larry Rosenman" 
>>>>> <ler at lerctr.org>
>>>>>> # cat send.stream | ssh root at home openssl md5
>>>>>> (stdin)= 9cd1d73ea8411f1c222bc90e7bea3d33
>>>>>> # uname -a
>>>>>> FreeBSD thebighonker.lerctr.org 8.3-STABLE FreeBSD 8.3-STABLE #54 
>>>>>> r247820: Mon Mar  4 18:08:11 CST 2013     
>>>>>> root at thebighonker.lerctr.org:/usr/obj/usr/src/sys/THEBIGHONKER  
>>>>>> amd64
>>>>>> #
>>>>>> # exec csh
>>>>>> root at borg:/home/ler # pwd
>>>>>> /home/ler
>>>>>> root at borg:/home/ler # ssh tbh "cat 
>>>>>> /home/ler/public_html/ZFS_RECV/send.stream | ssh 
>>>>>> root at home.lerctr.org openssl md5"
>>>>>> (stdin)= 9cd1d73ea8411f1c222bc90e7bea3d33
>>>>>> root at borg:/home/ler # uname -a
>>>>>> FreeBSD borg.lerctr.org 10.0-CURRENT FreeBSD 10.0-CURRENT #124 
>>>>>> r247826: Mon Mar  4 19:59:08 CST 2013     
>>>>>> root at borg.lerctr.org:/usr/obj/usr/src/sys/BORG-DTRACE  amd64
>>>>>> root at borg:/home/ler #
>>>>> When you run a command from ssh it will run as the target users 
>>>>> shell
>>>>> so for csh if there is something bad in .cshrc or iirc .profile 
>>>>> then
>>>>> you may get something output purely by logging in.
>>>>> So ensure you shell config's for the user on the target machine 
>>>>> are
>>>>> clean.
>>>>> You can confirm this by comparing the output from the following
>>>>> two commands run from the sending machine:-
>>>>> md5 send.stream
>>>>> cat send.stream | ssh root at home.lerctr.org "cat" |md5
>>>>> If these two commands produce different md5's then the login
>>>>> shell for root on home.lerctr.org is outputing something
>>>>> even when there is no terminal.
>>>>> You should be able to see what the shell is outputting using:-
>>>>> echo -n '' | ssh root at home.lerctr.org "cat" > 1.log
>>>>> After running this 1.log should be 0 bytes if its not your
>>>>> shell is definitely your problem and I'd start by looking
>>>>> at .cshrc and .profile for this user, or even the globals
>>>>> for these in /etc/
>>>>> 
>>>> I've done that, and get identical md5's.....
>>>> I'm at a loss.
>>> I believe what you ran was similar but not identical, if you
>>> check the quoting subtly different.
>>> I'm not sure that will have any effect but both of the above
>>> are worth running just to confirm.
>>> Regards
>>>    Steve
>> $ ^DConnection to thebighonker.lerctr.org closed.
>> $ ssh root at tbh
>> Last login: Tue Mar  5 06:51:46 2013 from cpe-72-182-19-1
>> FreeBSD 8.3-STABLE (THEBIGHONKER) #54 r247820: Mon Mar  4 18:08:11 
>> CST 2013
>> Welcome to FreeBSD!
>> root at thebighonker:~ # pwd
>> /root
>> root at thebighonker:~ # md5 /home/ler/public_html/ZFS_RECV/send
>> send-file-snap-list  send.stream          send.uname
>> root at thebighonker:~ # md5 /home/ler/public_html/ZFS_RECV/send.stream
>> MD5 (/home/ler/public_html/ZFS_RECV/send.stream) = 
>> 9cd1d73ea8411f1c222bc90e7bea3d33
>> root at thebighonker:~ # cat /home/ler/public_html/ZFS_RECV/send.stream 
>> |ssh root at home.lerctr.org "cat"|md5
>> 9cd1d73ea8411f1c222bc90e7bea3d33
>> root at thebighonker:~ #
>> 
>> looks like it's clean.....
> 
> So to summarise, just to make sure I've got everything clear so far:-
> 1. sending the stream over ssh from the source -> target results in
> the same md5 (from the test above)
yes
> 2. sending the stream over ssh from source -> target piped into
> zfs recv fails
yes, sometimes now (I need to see what happens from cron(8) tonight)
> 3. scp'ing the stream file from the source to the target and then
> importing it via zfs recv works
> 
yes
> Yes / No?
> 
> If so this is very strange indeed.
> 
> I'd like to confirm that adding the zfs recv to the picture isn't 
> changing
> the dynamic such that the stdin results are changing and thus 
> eliminating
> ssh as the source of any corruption to do this can you test:-
> md5 send.stream
> cat send.stream | ssh root at home.lerctr.org "tee tea.stream | zfs recv 
> <flags>"
> ssh root at home.lerctr.org "md5 tea.stream"
> 
Will try this with tonights stream(s) if it fails.

appreciate.....
>    Regards
>    Steve
> 
> ================================================
> This e.mail is private and confidential between Multiplay (UK) Ltd.
> and the person or entity to whom it is addressed. In the event of
> misdirection, the recipient is prohibited from using, copying,
> printing or otherwise disseminating it or any information contained in
> it.
> 
> In the event of misdirection, illegible or incomplete transmission
> please telephone +44 845 868 1337
> or return the E.mail to postmaster at multiplay.co.uk.

-- 
Larry Rosenman                     http://www.lerctr.org/~ler
Phone: +1 214-642-9640 (c)     E-Mail: ler at lerctr.org
US Mail: 430 Valona Loop, Round Rock, TX 78681-3893