Re: RFC reviews for ggate and hastd

On 14/09/2021 22:21, John-Mark Gurney wrote:
> Johannes Totz wrote this message on Mon, Sep 13, 2021 at 02:00 +0100:
>> On 09/09/2021 23:33, John-Mark Gurney wrote:
>>> Johannes Totz via freebsd-geom wrote this message on Thu, Sep 02, 2021 at 21:55 +0100:
>>>> (looks like gmane swallowed my 1st message, trying again)
>>>>
>>>> Hey folks,
>>>>
>>>> any ggate or hastd users here? I've got some code reviews for you.
>>>> Please take a look if you get a chance:
>>>>
>>>> https://reviews.freebsd.org/D31727
>>>> Fix potential out-of-bounds read in the geom-gate kernel module.
>>>>
>>>> https://reviews.freebsd.org/D31722
>>>> Dynamically alloc buffers in ggatec, instead of assuming a fixed size on
>>>> the stack.
>>>>
>>>> https://reviews.freebsd.org/D31709
>>>> Simple rc script to start ggated.
>>>
>>> I'll try to look at them.
>>>
>>> I've broken out the ggate code to: https://www.funkthat.com/gitea/jmg/ggate
>>
>> Nice, thanks!
>>
>> I noticed the http branch. One weekend toy project idea I wanted to get
>> around to was to write a ggated impl that talks to Backblaze.
> 
> I looked at the Backblaze B2 API, and I don't see a way to do partial
> updates of a file.  All the API that I see require you to upload the entire
> file, so I don't think that it'll work.
> 
> I abanded http as a solution, because of issues w/ WebDAV and partial
> updates and the IETF not being very sane about it:
> https://blog.sphere.chronosempire.org.uk/2012/11/21/webdav-and-the-http-patch-nightmare

Oh I was just gonna store each block as a separate file on the backend. 
No need to keep it all together as one huge image. Sure we'd end up with 
literally billions of small files. But I'd ignore that until it becomes 
an actual problem.

> Also, the fact that apache would randomly truncate a file didn't help
> either...
> 
>>> and have a number of fixes in that tree, one of them being having the
>>> server (ggated) be multi-threaded, to improve IOPs.  Another is dropping
>>> the setting of tcp buffer sizes to allow the dynamic buffer sizing code
>>> to work, allowing it to grow larger than 128kB for better perf on high
>>> latency links..
>>>
>>> In another branch, I have a version that runs the client over ssh using
>>> sftp (requires a modified sftp-server for best performance, and security)
>>> allowing for better security..
>>>
>>> The ssh branch does have an rc.d script for that.  I'll look at which one
>>> should be used, and adapting the man page that I've written for that one
>>> as well.
>>>
>>> Thanks for the help/work!
>