zfs send/recv invalid data

[Larry Rosenman]

On Wed, 6 Mar 2013, Ronald Klop wrote:

> On Wed, 06 Mar 2013 20:15:02 +0100, Larry Rosenman <ler at lerctr.org> wrote:
>
>> On 2013-03-06 12:59, Larry Rosenman wrote:
>>> On 2013-03-06 12:57, Martin Simmons wrote:
>>>>>>>>> On Wed, 06 Mar 2013 05:07:28 -0600, Larry Rosenman said:
>>>>> On 2013-03-06 04:56, Larry Rosenman wrote:
>>>>>> On 2013-03-06 04:51, Tom Evans wrote:
>>>>>>> On Wed, Mar 6, 2013 at 10:46 AM, Larry Rosenman <ler at lerctr.org>
>>>>>>> wrote:
>>>>>>>> So, you are correct that something(tm) is unclean about the ssh
>>>>>>>> path.
>>>>>>>> 
>>>>>>> And what is it that is added? Surely that is trivial to determine?
>>>>>>> echo hello > foo
>>>>>>> cat foo | ssh host cat
>>>>>>> Cheers
>>>>>>> Tom
>>>>>> it seems to be bytestream dependent, as this simple case works, and
>>>>>> the vast majority of the sends work, but the particular
>>>>>> case in the thread does NOT.
>>>>>> # echo hello >foo
>>>>>> # cat foo | ssh home cat
>>>>>> hello
>>>>>> #
>>>>> stranger:
>>>>> $ cd /tmp
>>>>> $ cat send.stream | ssh home openssl md5
>>>>> (stdin)= 9cd1d73ea8411f1c222bc90e7bea3d33
>>>>> $ openssl md5 send.stream
>>>>> MD5(send.stream)= 9cd1d73ea8411f1c222bc90e7bea3d33
>>>>> $
>>>>> so I'm not sure what is tripping it in the stream to a pipe case.
>>>> Is that running under the same uid?  The script you posted earlier runs 
>>>> the
>>>> pipe under ssh root at tbh.lerctr.org.
>>>> __Martin
>>> Ooo, good point.  Would csh vs. sh make a difference?
>> Doesn't seem to make a difference....
>> 
>> # cat send.stream | ssh root at home openssl md5
>> (stdin)= 9cd1d73ea8411f1c222bc90e7bea3d33
>> # uname -a
>> FreeBSD thebighonker.lerctr.org 8.3-STABLE FreeBSD 8.3-STABLE #54 r247820: 
>> Mon Mar  4 18:08:11 CST 2013 
>> root at thebighonker.lerctr.org:/usr/obj/usr/src/sys/THEBIGHONKER  amd64
>> #
>> # exec csh
>> root at borg:/home/ler # pwd
>> /home/ler
>> root at borg:/home/ler # ssh tbh "cat 
>> /home/ler/public_html/ZFS_RECV/send.stream | ssh root at home.lerctr.org 
>> openssl md5"
>> (stdin)= 9cd1d73ea8411f1c222bc90e7bea3d33
>> root at borg:/home/ler # uname -a
>> FreeBSD borg.lerctr.org 10.0-CURRENT FreeBSD 10.0-CURRENT #124 r247826: Mon 
>> Mar  4 19:59:08 CST 2013 
>> root at borg.lerctr.org:/usr/obj/usr/src/sys/BORG-DTRACE  amd64
>> root at borg:/home/ler #
>> 
>
>
> Hi,
>
> My script to send/rcv locally has a commented mbuffer in it because I had 
> trouble with it in the past.
>
> zfs send $SEND_ARGS $VOL_FROM@$PREFIX$NEW \
>       | zfs receive -F $VOL_TO
> #       | mbuffer -q | zfs receive -F $VOL_TO
>
> <guess>
> Maybe zfs receive expects a certain size of blocks of data and |ssh (or in my 
> case |mbuffer) changes that. This is all a wild guess. More debugging should 
> tell the difference, but it might give a direction to look for.
> </guess>
>
> The error "invalid backup stream" can be given at a couple of places. Can you 
> use a debugger or ktrace or something else to get more information about what 
> is happening?
>
> $ 20:32:24 ronald at sjakie 
> [/usr/src/cddl/contrib/opensolaris/lib/libzfs/common]
> grep EZFS_BADSTREAM *
> libzfs.h:       EZFS_BADSTREAM,         /* bad backup stream */
> libzfs_sendrecv.c:              return (zfs_error(hdl, EZFS_BADSTREAM, 
> dgettext(TEXT_DOMAIN,
> libzfs_sendrecv.c:                      error = zfs_error(hdl, 
> EZFS_BADSTREAM, errbuf);
> libzfs_sendrecv.c:              error = zfs_error(hdl, EZFS_BADSTREAM, 
> errbuf);
> libzfs_sendrecv.c:              error = zfs_error(hdl, EZFS_BADSTREAM, 
> errbuf);
> libzfs_sendrecv.c:              error = zfs_error(hdl, EZFS_BADSTREAM, 
> errbuf);
> libzfs_sendrecv.c:                              return (zfs_error(hdl, 
> EZFS_BADSTREAM, errbuf));
> libzfs_sendrecv.c:                      return (zfs_error(hdl, 
> EZFS_BADSTREAM, errbuf));
> libzfs_sendrecv.c:                      return (zfs_error(hdl, 
> EZFS_BADSTREAM, errbuf));
> libzfs_sendrecv.c:                      (void) zfs_error(hdl, EZFS_BADSTREAM, 
> errbuf);
> libzfs_sendrecv.c:                      (void) zfs_error(hdl, EZFS_BADSTREAM, 
> errbuf);
> libzfs_sendrecv.c:              return (zfs_error(hdl, EZFS_BADSTREAM, 
> errbuf));
> libzfs_sendrecv.c:              return (zfs_error(hdl, EZFS_BADSTREAM, 
> errbuf));
> libzfs_sendrecv.c:              return (zfs_error(hdl, EZFS_BADSTREAM, 
> errbuf));
> libzfs_util.c:  case EZFS_BADSTREAM:

Hrm.  Anyone have an idea on what to do with ktrace or dtrace or gdb or whatever
to debug this?

I'm at a loss on where to start on it :(


-- 
Larry Rosenman                     http://www.lerctr.org/~ler
Phone: +1 512-248-2683                 E-Mail: ler at lerctr.org
US Mail: 430 Valona Loop, Round Rock, TX 78681-3893