zfs-send/receive between FreeBSD10 and SmartOS (Illumos) fails.

[John Terrell]

Ok, I’ve confirmed this isn’t a true ZFS send/receive issue - it is, in fact, an environment issue here.   There’s a switch in my environment that a few times a day goes put to lunch (disconnects) for 5-10 seconds.   This disconnection was enough to trip up the send/receive process but not enough to disrupt the SSH session that initiated the transfer.   

Thanks for all of the help and sorry to randomize the discussion.

On Oct 1, 2014, at 6:57 AM, John Terrell <john.terrell at gmail.com> wrote:

> Eureka!   I think I’ve found the issue and it’s nothing to do with ZFS - it’s an infrastructure problem.   I was running another test last night and just happened to be watching about two hours into the test when the Mac I use to connect to the network suddenly (and unexpectedly) lost connectivity for about five seconds.   When it came back, the SSH connection was intact but the ZFS transfer in progress immediately failed.   I’m not sure if this is a DHCP lease re-aquisition or what but to see if this is indeed the problem, I gave the Mac a manual IP address and started the transfer over night.   As of now, it’s been running for about 7 hours without issue.  I’m not quite ready to call it solved until this transfer completes successfully but this is the furthest it’s gotten.
> 
> 
> On Sep 30, 2014, at 8:02 PM, John Terrell <john.terrell at gmail.com> wrote:
> 
>> Sure.   On the SmartOS (receiving) side:
>> 
>> [root at smartos /]# zpool get all zones | grep feature@
>> zones  feature at async_destroy          enabled                        local
>> zones  feature at empty_bpobj            active                         local
>> zones  feature at lz4_compress           active                         local
>> zones  feature at multi_vdev_crash_dump  enabled                        local
>> zones  feature at spacemap_histogram     active                         local
>> zones  feature at enabled_txg            active                         local
>> zones  feature at hole_birth             active                         local
>> zones  feature at extensible_dataset     enabled                        local
>> zones  feature at embedded_data          active                         local
>> zones  feature at bookmarks              enabled                        local
>> zones  feature at filesystem_limits      enabled                        local
>> 
>> …on the FreeBSD 10 (sending) side:
>> 
>> [root at freebsd /]# zpool get all tank | grep feature@
>> tank  feature at async_destroy          enabled                        local
>> tank  feature at empty_bpobj            active                         local
>> tank  feature at lz4_compress           active                         local
>> tank  feature at multi_vdev_crash_dump  enabled                        local
>> 
>> 
>> So, the FreeBSD side is a subset of the SmartOS side.
>> 
>> 
>> On Sep 30, 2014, at 7:51 PM, Rich <rincebrain at gmail.com> wrote:
>> 
>>> It's possible that your feature flags and the receiving end's aren't
>>> compatible - can you show us a zfs get all tank on both sides?
>>> 
>>> - Rich
>>> 
>>> On Tue, Sep 30, 2014 at 10:39 PM, John Terrell <john.terrell at gmail.com> wrote:
>>>> By the way, here's another report that sounds similar to mine (though on Linux):
>>>> 
>>>>        http://stackoverflow.com/questions/19754375/why-does-zsh-send-fail-midway
>>>> 
>>>> His repro doesn't even use ssh (appears to be completely local).
>>>> 
>>>> 
>>>> On Sep 30, 2014, at 6:41 PM, John Terrell <john.terrell at gmail.com> wrote:
>>>> 
>>>>> Sorry for the tardy response (busy at work. :)).   Here's the command and output (run from the SmartOS box):
>>>>> 
>>>>> ssh -c arcfour256 root at freebsdnas zfs send -R tank at daily20140920 | zfs receive -v -F zones/tank
>>>>> 
>>>>> The output (after running for a long time xferring data):
>>>>> 
>>>>> receiving full stream of tank at daily20140920 into zones/tank at daily20140920
>>>>> received 1.35GB stream in 15 seconds (91.9MB/sec)
>>>>> receiving full stream of tank/photography at daily20140920 into zones/tank/photography at daily20140920
>>>>> Read from remote host nas00: Connection reset by peer
>>>>> cannot receive new filesystem stream: invalid backup stream
>>>>> 
>>>>> 
>>>>> Is it possible the command is simply timing out for some reason and closing the connection?
>>>>> 
>>>>> 
>>>>> On Sep 26, 2014, at 3:04 PM, Will Andrews <will at firepipe.net> wrote:
>>>>> 
>>>>>> What do the zfs commands print? The -v option prints some metadata that might help diagnose the problem.
>>>>>> 
>>>>>> --Will.
>>>>>> 
>>>>>> On Monday, September 22, 2014, John Terrell <john.terrell at gmail.com> wrote:
>>>>>> Hello all, I'm trying to replicate one of the ZFS filesystems that lives on my FreeBSD 10 NAS to a SmartOS (Illumos based) server.   The transfer dies at about the 1.7TB (of almost 4TB) mark indicating:
>>>>>> 
>>>>>> "cannot receive incremental stream: invalid backup stream"
>>>>>> 
>>>>>> The stream being sent is not incremental so I'm not sure what the receiver is trying to do.    Here's the command I'm using to transfer (executed on the SmartOS machine):
>>>>>> 
>>>>>> ssh root at fbsd10 zfs send -v -R tank at daily20140920 | zfs receive -v tank
>>>>>> 
>>>>>> I've also tried to use mbuffer as the transport interface (removing SSH from the equation) and it has the same result.
>>>>>> 
>>>>>> Is the possibly an incompatibility between FreeBSD10 and SmartOS ZFS implementations?
>>>>>> _______________________________________________
>>>>>> freebsd-fs at freebsd.org mailing list
>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
>>>>>> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"
>>>>> 
>>>> 
>>>> _______________________________________________
>>>> freebsd-fs at freebsd.org mailing list
>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
>>>> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"
>> 
>