From nobody Sun Aug 22 23:54:39 2021 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id BD5C3179051B for ; Sun, 22 Aug 2021 23:54:51 +0000 (UTC) (envelope-from ggm@algebras.org) Received: from mail-lf1-x133.google.com (mail-lf1-x133.google.com [IPv6:2a00:1450:4864:20::133]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4GtC0W4Sp5z3CVT for ; Sun, 22 Aug 2021 23:54:51 +0000 (UTC) (envelope-from ggm@algebras.org) Received: by mail-lf1-x133.google.com with SMTP id u22so34102291lfq.13 for ; Sun, 22 Aug 2021 16:54:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=algebras-org.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=XdFsrim2L3LuBwNh362BPW3tLWGSnm6KW8cP86fTv+A=; b=f8APggIBAjzaGLYM8v1KF0JvCXTOUGhl3/dhd+xSwWgIW5flWpWV9DOIpYyBGB2cBN qoUBx8DgWPKcBFV4OtjS6WCkgAcxdZSkY8XNb1yZVepnxSA9Pn+ElqcsVeX49QIczjhD XRwiZG4DqTs4TGOcU5P45cfNGp5dlIdnSFb5E+T2LIL/vCLX9XH5mZo4VazeofHbIaal +JSVAWuRORALtHT8XqZb5rFg0NNq6nd6pvyXsqUgGFIgHeN/BJ0foT/XjayWmleByxWH kC9Ig9lUwJwQG8FSzfvgtuIiiZOOIh06dA6SOM4Yk2rYhLo9EjyWnGSengEFGoK3kYQY AN/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=XdFsrim2L3LuBwNh362BPW3tLWGSnm6KW8cP86fTv+A=; b=Th1Jp6MjTqmb/FPvFuepjpIbdoEFO4ImjeT5HfpFXjzrKWSPDwaeMuIxpQ2GQXI7dk Kvdci3VqBmEVOakjWJo5HPFoppa3V3zMkeUKoWiW7dRzG1pYcd3QHqvF1QBq4Mxd3YJA R496u7xuJ5si/9oRZ/CPnbxQz5vmbY+7oZyGkwg24pjBGIxcRHwlwjY9L8IBMsxemgxf zh4e1p9dXAcjMg0f02mEayDkXuOQKpKVIn9a73Q8v/eD4xdQs7KcFR6TyUhWNpJdZaR4 rwBFu703Bz4HJKJ4EeRTBPS8BvSoOhbaaNrlbbeXh4OlNr2q0A+gratqojlWmmJoXBfu dxfw== X-Gm-Message-State: AOAM530pCCO3jZOpuDOlEvRlGWRU9aC01e9OEi/M+q6P194uhIgsgK9Y Z60bwtuRrgQd1ZJjDmMvz8FTEc5U8stI2iD375J3nQ== X-Google-Smtp-Source: ABdhPJwMS6T5gscGAdu5wpTlqamoUUDNH8GNeieIe4tQGEIw3bwFPb/0XxJThHdWZsCH6HKDy7C1XEuOJ/yVv59HFy0= X-Received: by 2002:a05:6512:16a0:: with SMTP id bu32mr23133507lfb.322.1629676490187; Sun, 22 Aug 2021 16:54:50 -0700 (PDT) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 References: <023225AD-2A97-47C5-9FE4-3ABF1BFD66F1@gmx.com> In-Reply-To: From: George Michaelson Date: Mon, 23 Aug 2021 09:54:39 +1000 Message-ID: Subject: Re: ZFS on high-latency devices To: Alan Somers Cc: Ben RUBSON , freebsd-fs Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 4GtC0W4Sp5z3CVT X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-ThisMailContainsUnwantedMimeParts: N I don't think its sensible to mesh long-delay file constructs into a pool. Maybe there is a model which permits this but I think higher abstractions like CEPH may be a better fit for the kind of distributed filestore this implies. I use ZFS send/receive to make "clones" of a zpool/zfs structure exist, and they are not bound in the delay problem for live FS delivery: you use mbuffer to make the transport of the entire ZFS datastate more effective. A long time ago, 25+ years ago I ran NFS over X25, as well as SMB. (ip over X25) and it was very painful. Long-haul, long delay networks are not a good fit for direct FS semantics of read/write if you care about speed. There isn't enough cacheing in the world to make the distance fully transparent. When I have to use FS-over- now it's typically to mount virtual ISO images for console-FS recovery of a host, and its bad enough over 150ms delay fibre links not to want to depend on it beyond the bootstrap phase. -G On Mon, Aug 23, 2021 at 9:48 AM Alan Somers wrote: > > mbuffer is not going to help the OP. He's trying to create a pool on top= of a networked block device. And if I understand correctly, he's connecti= ng over a WAN, not a LAN. ZFS will never achieve decent performance in suc= h a setup. It's designed as a local file system, and assumes it can quickl= y read metadata off of the disks at any time. The OP's best option is to g= o with "a": encrypt each dataset and send them with "zfs send --raw". I do= n't know why he thinks that it would be "very difficult". It's quite easy,= if he doesn't care about old snapshots. Just: > > $ zfs create pool/new_dataset > $ cp -a pool/old_dataset/* pool/new_dataset/ > > -Alan > > On Sun, Aug 22, 2021 at 5:40 PM George Michaelson wrot= e: >> >> I don't want to abuse the subject line too much, but I can highly >> recommend the mbuffer approach, I've used this repeatedly, BSD-BSD and >> BSD-Linux. It definitely feels faster than SSH, since the 'no cipher' >> options were removed, and in the confusion of the HPC buffer changes. >> But, its not crypted on-the-wire. >> >> Mbuffer tuning is a bit of a black art: it would help enormously if >> there was some guidance on this, and personally I've never found the >> mbuffer -v option to work well: I get no real sense of how full or >> empty the buffer "is" or, if the use of sendmsg/recvmsg type buffer >> chains is better or worse. >> >> -G >> >> On Fri, Aug 20, 2021 at 6:19 PM Ben RUBSON wrote: >> > >> > > On 19 Aug 2021, at 11:37, Peter Jeremy wrote: >> > > >> > > (...) or a way to improve throughput doing "zfs recv" to a pool with= a high RTT. >> > >> > You should use zfs send/receive through mbuffer, which will allow to s= ustain better throughput over high latency links. >> > Feel free to play with its buffer size parameters to find the better s= ettings, depending on your link characteristics. >> > >> > Ben >> > >> > >>