From nobody Thu Feb 23 19:15:32 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PN2mc5h9dz3sdCv for ; Thu, 23 Feb 2023 19:15:44 +0000 (UTC) (envelope-from bsdunix44@gmail.com) Received: from mail-pj1-x102d.google.com (mail-pj1-x102d.google.com [IPv6:2607:f8b0:4864:20::102d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PN2mc3lw2z43Kc for ; Thu, 23 Feb 2023 19:15:44 +0000 (UTC) (envelope-from bsdunix44@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-pj1-x102d.google.com with SMTP id cp7-20020a17090afb8700b0023756229427so360227pjb.1 for ; Thu, 23 Feb 2023 11:15:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=eSnL7H2JqspZngpdyxwyjiXUlI+P21ydRQwHKrXneO4=; b=UOMF8KoXyycLfzXJ3OSSy3nD8qrFhRj14R5MMPQBmdMmEr3iHecrKsb8j6a2JD6nQ5 cdW9uPP5EpuWCF+WUfuEc8JCm6gqZrzYgY+BEGXf3W9uMOkRQ4VBNNfzfqgcLe6W/Llc b9CcRnZiQQ5y3o+ex9lpujbkwGCCRiM/qbY8Be1Se9irdt6SpsNVquUHn9MDzDNA6BrI DEz3KlWqbS8mwbRemk3SwV6zr61bLSC5WGn/h56NPS8O8hsaKILlSpuDbI2kQGioPNd4 NFdcyBjg8cncZi9I1s7iL1KVW61OweFrut5Kuicm7g55KlzUd+9NjVMZRcUL2ia3PpKY g8FA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=eSnL7H2JqspZngpdyxwyjiXUlI+P21ydRQwHKrXneO4=; b=Oea4SE5tRERhJXL+jccC6jTSQrV0bMMrBxpz5PiBbVJhxNDLur8zngjvbbtYCxeOM8 Fw1Hj2lqHQnplIlC+dKgz+jSG2PLXyTTHkDeZbgkNaF0vt4YXX54Cp+FWUCcJYbWTmKF ab8VNRfZuyQ4uoaGMChyZvrn+h/4P0MiuA83pyOlDwDwowPr5V8EwxXbmofT5eSrDRJ+ 9BGjjso2LP7iygAYvcXM4OPLGz1uw5PUJS5oDXDDLy1CTo7CzGMGj6njRKN5lKv5TXLK jzeYRyYGY3lHtI2bZEQ4+htuTYOUM59CN+0DjFLBord80mmEQx4xAhnDVBpg0SKIhaq+ aA9Q== X-Gm-Message-State: AO0yUKVeKAz6zEsrmTPJeiVEKPfwkBxx/4HYKjIAZqYsQregJsAyFCG1 q/NRirYGP72VxpdawHOkUGH3H8rED2gYPhGg+HFcSQFd X-Google-Smtp-Source: AK7set/bETShe/DgE0Q+6rbl9v8G8/11PnymBWF0ByoTCnuHWniOPAM6WsC6FqgJnF9ritoEP2imYUAqrRekCD+ixoY= X-Received: by 2002:a17:903:3291:b0:19a:9f86:adab with SMTP id jh17-20020a170903329100b0019a9f86adabmr2031687plb.7.1677179743156; Thu, 23 Feb 2023 11:15:43 -0800 (PST) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 References: <866d6937-a4e8-bec3-d61b-07df3065fca9@sentex.net> <1031e2b0-b245-1dc6-a499-8f4da3796543@quip.cz> <46455168-d7f1-6ca9-ad2f-9bcd3359e0f3@sentex.net> <78c78aec-a34b-f188-ef96-8ced9a1eda35@quip.cz> <741387429.91447.1677122934622@ichabod.co-bxl> In-Reply-To: <741387429.91447.1677122934622@ichabod.co-bxl> From: Chris Watson Date: Thu, 23 Feb 2023 13:15:32 -0600 Message-ID: Subject: Re: speeding up zfs send | recv (update) To: Sysadmin Lists Cc: Freddie Cash , freebsd-fs Content-Type: multipart/alternative; boundary="000000000000d0c06905f562d633" X-Rspamd-Queue-Id: 4PN2mc3lw2z43Kc X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --000000000000d0c06905f562d633 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable [Sorry miroslav, I hit send without checking the To: this was meant to be public] I=E2=80=99m a bit late, but I mentioned this to someone on this thread priv= ately, I=E2=80=99m curious why =E2=80=98spiped=E2=80=99 hasn=E2=80=99t been mentio= ned in this thread. I=E2=80=99ve seen everything from VPN=E2=80=99s to nc. VPNs would be, imo, grossly unwarranted/massively overly complex/hard to secure just to simply have a secure pipe for doing ZFS send|recv. Simply configuring an spiped PtP pipe between A and B seems the simplest, most secure, performant option here. At least considering all the other options tossed out in this thread. No one=E2=80=99s using spiped? O.o Thoughts? Has anyone compared ssh to spiped regarding overhead and throughput in this scenario? Chris On Wed, Feb 22, 2023 at 9:29 PM Sysadmin Lists wrote: > > On Feb 22, 2023 at 1:43 PM, Freddie Cash wrote: > > [Sorry for top part, GMail sucks for replies.] > > If this is a LAN or private WAN where you trust the network, piping the > send stream through netcat will remove ssh from the equation. > > That's what we switched to using once it became almost impossible to get > the "none" cipher working with ssh on FreeBSD. > > We use ssh to connect to the remote server and enable a netcat listener o= n > port X, then pipe the send through netcat to the remote system on port X. > That way it's logged and uses ssh for authentication. > > We easily saturate gigabit links between our ZFS systems using netcat. > > > > Cheers, > Freddie > > Typos due to smartphone keyboard. > > On Wed., Feb. 22, 2023, 1:31 p.m. Miroslav Lachman, <000.fbsd@quip.cz> > wrote: > > On 22/02/2023 22:08, mike tancsa wrote: > > On 2/22/2023 4:03 PM, Miroslav Lachman wrote: > >> Interresting numbers. I think I am the only one who get best speed > >> with chacha20-poly1305@openssh.com > >> > >> > >> It seems the speed of SSH is limited by single core performance which > >> is very poor on this machine (Intel(R) Pentium(R) Dual CPU E2160). > >> Even if CPU has 50% idle, ssh runs on 99.8% of single core. > > > > The CPU I have has > > aesni0: on motherboard > > > > which probably helps. > > That explains it > aesni0: No AES or SHA support. > > >> I know there were some HPN patches to ssh, beside that is there any > >> option I can try to use less CPU? > >> > >> I will play with cpuset to pin ssh on one core and everything else on > >> the other core. > > > > It looks like you are running into a CPU bottleneck TBH > > Yes. Pinning on cores with cpuset helps a bit (about +3MiB/s) but > without some tweaks on ssh I will not gain more speed :( > > Thank you for your help! > > Miroslav Lachman > > > > You could pipe the stream through an encrypting program before piping to > netcat, then decrypt on the recieving end. > > $ zfs send | crypt | netcat ipaddr 2222 > $ netcat -vl 2222 | crypt | zfs recv > > I don't know if zfs can handle that, but worth a try. > > $ man crypt > The enigma utility, also known as crypt is a very simple encryption > program, working on a =E2=80=9Csecret-key=E2=80=9D basis. It operat= es as a filter, > i.e., > it encrypts or decrypts a stream of data from standard input, and > writes > the result to standard output. Since its operation is fully > symmetrical, > feeding the encrypted data stream again through the engine (using th= e > same secret key) will decrypt it. > > > -- Sent with https://mailfence.com Secure and private email --000000000000d0c06905f562d633 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
[Sorry miroslav, I hit send without ch= ecking the To: this was meant to be public]=C2=A0

I=E2=80=99m a bit lat= e, but I mentioned this to someone on this thread privately, I=E2=80=99m cu= rious why =E2=80=98spiped=E2=80=99 hasn=E2=80=99t been mentioned in this th= read. I=E2=80=99ve seen everything from VPN=E2=80=99s to nc. VPNs would be,= imo, grossly unwarranted/massively overly complex/hard to secure just to s= imply have a secure pipe for doing ZFS send|recv.=C2=A0

Simply configuring an spiped P= tP pipe between A and B seems the simplest, most secure, performant option = here. At least considering all the other options tossed out in this thread.= =C2=A0

No o= ne=E2=80=99s using spiped? O.o

Thoughts?=C2=A0

Has anyone compared ssh to spiped regarding overhe= ad and throughput in this scenario?
Chris

On Wed, Feb 22, 202= 3 at 9:29 PM Sysadmin Lists <sysadmin.lists@mailfence.com> wrote:
=

= On Feb 22, 2023 at 1:43 PM, Freddie Cash <= fjwcash@gmail.com> wrote:
[Sorry for top part, GMail sucks for replies.]
=
If this= is a LAN or private WAN where you trust the network, piping the send strea= m through netcat will remove ssh from the equation.

That's what we switched= to using once it became almost impossible to get the "none" ciph= er working with ssh on FreeBSD.

We use ssh to connect to the remote server= and enable a netcat listener on port X, then pipe the send through netcat = to the remote system on port X. That way it's logged and uses ssh for a= uthentication.

We easily saturate gigabit links between our ZFS systems us= ing netcat.



Cheers,
Freddie
=
Typos due to smartphone keyboard.

On Wed., Feb. 22, 2023, 1:31 p.m. Miro= slav Lachman, <000.fbsd@quip.cz> wro= te:
On 22/02/2023 22:08, mike t= ancsa wrote:
> On 2/22/2023 4:03 PM, Miroslav Lachman wrote:
>> Interresting numbers. I think I am the only one who get best speed=
>> with = chacha20-poly1305@openssh.com
>>
>>
>> It seems the speed of SSH is limited by single core performance wh= ich
>> is very poor on this machine (Intel(R) Pentium(R) Dual=C2=A0 CPU E= 2160).
>> Even if CPU has 50% idle, ssh runs on 99.8% of single core.
>
> The CPU I have has
> aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS> on motherboard=
>
> which probably helps.

That explains it
aesni0: No AES or SHA support.

>> I know there were some HPN patches to ssh, beside that is there an= y
>> option I can try to use less CPU?
>>
>> I will play with cpuset to pin ssh on one core and everything else= on
>> the other core.
>
> It looks like you are running into a CPU bottleneck TBH

Yes. Pinning on cores with cpuset helps a bit (about +3MiB/s) but
without some tweaks on ssh I will not gain more speed :(

Thank you for your help!

Miroslav Lachman



You could = pipe the stream through an encrypting program before piping to
netcat, then decrypt on the= recieving end.

$ zfs sen= d | crypt | netcat ipaddr 2222
$ netcat -vl 2222 | crypt | zfs recv

I don't know if zfs can handle that, but wo= rth a try.
$ man cr= ypt
=C2=A0 =C2=A0 The enigma utili= ty, also known as crypt is a very simple encryption
=C2=A0 =C2=A0 =C2=A0program, working o= n a =E2=80=9Csecret-key=E2=80=9D basis.=C2=A0 It operates as a filter, i.e.= ,
=C2=A0 =C2=A0= =C2=A0it encrypts or decrypts a stream of data from standard input, and wr= ites
=C2=A0 =C2= =A0 =C2=A0the result to standard output.=C2=A0 Since its operation is fully= symmetrical,
= =C2=A0 =C2=A0 =C2=A0feeding the encrypted data stream again through the eng= ine (using the
= =C2=A0 =C2=A0 =C2=A0same secret key) will decrypt it.


--=20 Sent with https://mailf= ence.com =20 Secure and private email --000000000000d0c06905f562d633--