From nobody Sun Sep 19 16:27:40 2021 X-Original-To: freebsd-geom@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 89B8217CEBC2 for ; Sun, 19 Sep 2021 16:27:51 +0000 (UTC) (envelope-from jo@bruelltuete.com) Received: from email.jo-t.de (seppel.jo-t.de [45.132.244.126]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4HCClp34Ttz4vcj for ; Sun, 19 Sep 2021 16:27:50 +0000 (UTC) (envelope-from jo@bruelltuete.com) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=bruelltuete.com; s=bruelltuete18a; t=1632068792; bh=AO4uyn+04wrtc/9vdBah77g/Q2WeCLV8qVw8UBdOgBU=; h=Message-ID:Date:MIME-Version:Subject:To:References:From:From; b=QbFoySuqyZI31NTNKn0v59cuuFzGVttcSe/IoIsY3+rWyd7v3qJwemS0d9OxQMmBS ZDCV3QrKjR+73uIqjYiFLk1JnXQFv766jS6w1BxosBCvvedTCdVxQ4Qhj1bJ9A7qNz 3qFSKw1ts6TZuJ/Geiu3iks12tgYfDxMt4NSDbAZr6mgTGSdUTCgEFTKlmqtjVyJ9V RB8QN2qdEIxA+XjnAYDZjjEl7Egi0e4gYZf9g57UVssil9K+T3kV6WVu/OodU51z6v aWybnUhn/2lG/NRBXXESa11m8ijSQ4vjtjsWuN/ZlvQuStgGlPjvc1uiA1M7jkiQRL 50hCZYGZjAkqw== Message-ID: <4f79999f-9a07-531b-c8db-eb78b273b71f@bruelltuete.com> Date: Sun, 19 Sep 2021 17:27:40 +0100 List-Id: GEOM-specific discussions and implementations List-Archive: https://lists.freebsd.org/archives/freebsd-geom List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-geom@freebsd.org MIME-Version: 1.0 Subject: Re: RFC reviews for ggate and hastd Content-Language: en-GB To: freebsd-geom@FreeBSD.org References: <20210909223355.GE5928@funkthat.com> <31ac6be7-e438-00d0-b63a-b27b09ed5d58@bruelltuete.com> <20210914212137.GI5928@funkthat.com> Cc: jmg@funkthat.com In-Reply-To: <20210914212137.GI5928@funkthat.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4HCClp34Ttz4vcj X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bruelltuete.com header.s=bruelltuete18a header.b=QbFoySuq; dmarc=pass (policy=reject) header.from=bruelltuete.com; spf=pass (mx1.freebsd.org: domain of jo@bruelltuete.com designates 45.132.244.126 as permitted sender) smtp.mailfrom=jo@bruelltuete.com X-Spamd-Result: default: False [-4.00 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[bruelltuete.com:s=bruelltuete18a]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[bruelltuete.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[bruelltuete.com,reject]; NEURAL_HAM_SHORT(-1.00)[-0.997]; RCVD_COUNT_ZERO(0.00)[0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:197540, ipnet:45.132.244.0/22, country:DE]; MID_RHS_MATCH_FROM(0.00)[] Reply-To: jo@bruelltuete.com From: Johannes Totz via freebsd-geom X-Original-From: Johannes Totz X-ThisMailContainsUnwantedMimeParts: N On 14/09/2021 22:21, John-Mark Gurney wrote: > Johannes Totz wrote this message on Mon, Sep 13, 2021 at 02:00 +0100: >> On 09/09/2021 23:33, John-Mark Gurney wrote: >>> Johannes Totz via freebsd-geom wrote this message on Thu, Sep 02, 2021 at 21:55 +0100: >>>> (looks like gmane swallowed my 1st message, trying again) >>>> >>>> Hey folks, >>>> >>>> any ggate or hastd users here? I've got some code reviews for you. >>>> Please take a look if you get a chance: >>>> >>>> https://reviews.freebsd.org/D31727 >>>> Fix potential out-of-bounds read in the geom-gate kernel module. >>>> >>>> https://reviews.freebsd.org/D31722 >>>> Dynamically alloc buffers in ggatec, instead of assuming a fixed size on >>>> the stack. >>>> >>>> https://reviews.freebsd.org/D31709 >>>> Simple rc script to start ggated. >>> >>> I'll try to look at them. >>> >>> I've broken out the ggate code to: https://www.funkthat.com/gitea/jmg/ggate >> >> Nice, thanks! >> >> I noticed the http branch. One weekend toy project idea I wanted to get >> around to was to write a ggated impl that talks to Backblaze. > > I looked at the Backblaze B2 API, and I don't see a way to do partial > updates of a file. All the API that I see require you to upload the entire > file, so I don't think that it'll work. > > I abanded http as a solution, because of issues w/ WebDAV and partial > updates and the IETF not being very sane about it: > https://blog.sphere.chronosempire.org.uk/2012/11/21/webdav-and-the-http-patch-nightmare Oh I was just gonna store each block as a separate file on the backend. No need to keep it all together as one huge image. Sure we'd end up with literally billions of small files. But I'd ignore that until it becomes an actual problem. > Also, the fact that apache would randomly truncate a file didn't help > either... > >>> and have a number of fixes in that tree, one of them being having the >>> server (ggated) be multi-threaded, to improve IOPs. Another is dropping >>> the setting of tcp buffer sizes to allow the dynamic buffer sizing code >>> to work, allowing it to grow larger than 128kB for better perf on high >>> latency links.. >>> >>> In another branch, I have a version that runs the client over ssh using >>> sftp (requires a modified sftp-server for best performance, and security) >>> allowing for better security.. >>> >>> The ssh branch does have an rc.d script for that. I'll look at which one >>> should be used, and adapting the man page that I've written for that one >>> as well. >>> >>> Thanks for the help/work! >