From nobody Fri Feb 24 22:45:15 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PNlN15GWnz3v2WG for ; Fri, 24 Feb 2023 22:45:21 +0000 (UTC) (envelope-from sysadmin.lists@mailfence.com) Received: from wilbur.contactoffice.com (wilbur.contactoffice.com [212.3.242.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PNlN043mwz47vX for ; Fri, 24 Feb 2023 22:45:20 +0000 (UTC) (envelope-from sysadmin.lists@mailfence.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=mailfence.com header.s=20210208-e7xh header.b=WboHlsBo; spf=pass (mx1.freebsd.org: domain of sysadmin.lists@mailfence.com designates 212.3.242.68 as permitted sender) smtp.mailfrom=sysadmin.lists@mailfence.com; dmarc=pass (policy=quarantine) header.from=mailfence.com Received: from fidget.co-bxl (fidget.co-bxl [10.2.0.33]) by wilbur.contactoffice.com (Postfix) with ESMTP id 2230C1A1B; Fri, 24 Feb 2023 23:45:18 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1677278718; s=20210208-e7xh; d=mailfence.com; i=sysadmin.lists@mailfence.com; h=Date:From:To:Cc:Message-ID:In-Reply-To:References:Subject:MIME-Version:Content-Type; l=15015; bh=EC0i/VAxrIYv/ZhW5nn2AYVOIrEMShMKTWPgmQ9oWR8=; b=WboHlsBo5vwT9pqhVhQ1dzjRguPNk3cSes5mSI0wMqudIJXTN8ybwyP8hk7dOgMV if96HidVa3MwONroY7P7v07bTUpF1u02vSkBo/QSKHzWXkX5XqOjsU7CPAAu1NK+SK9 2JWRyBYOFqYrP9+G2yQ7qMCMkpuMI/OoHlQ7nc+ok7vDCyjSShUhXKt9tyzoW16gyXr T7MO6Ug77eXvmecFaMWNGTzcWlYN/Wm3QkdkjGuNtzkUP9J/Mf4y2iKmLJEeuXcrNgE Vunqj4KMSOXpMqhynvX9aZvIjXE5EfwsBGCn0CaAEyWBcbHn+SLiG/4FViKmVO49X+M Sq9vAO1cPQ== Date: Fri, 24 Feb 2023 23:45:15 +0100 (CET) From: Sysadmin Lists To: freebsd-fs Cc: Chris Watson Message-ID: <1290947438.348129.1677278715319@fidget.co-bxl> In-Reply-To: References: <866d6937-a4e8-bec3-d61b-07df3065fca9@sentex.net> <1031e2b0-b245-1dc6-a499-8f4da3796543@quip.cz> <46455168-d7f1-6ca9-ad2f-9bcd3359e0f3@sentex.net> <78c78aec-a34b-f188-ef96-8ced9a1eda35@quip.cz> <741387429.91447.1677122934622@ichabod.co-bxl> Subject: Re: speeding up zfs send | recv (update) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_348126_1685684535.1677278715318" X-Mailer: ContactOffice Mail X-ContactOffice-Account: com:312482426 X-Spamd-Result: default: False [-4.08 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.986]; DMARC_POLICY_ALLOW(-0.50)[mailfence.com,quarantine]; R_DKIM_ALLOW(-0.20)[mailfence.com:s=20210208-e7xh]; R_SPF_ALLOW(-0.20)[+ip4:212.3.242.64/26]; RCVD_IN_DNSWL_LOW(-0.10)[212.3.242.68:from]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; XM_UA_NO_VERSION(0.01)[]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ARC_NA(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[mailfence.com:+]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_SOME(0.00)[]; ASN(0.00)[asn:10753, ipnet:212.3.242.64/26, country:US]; FREEMAIL_CC(0.00)[gmail.com] X-Rspamd-Queue-Id: 4PNlN043mwz47vX X-Spamd-Bar: ---- X-ThisMailContainsUnwantedMimeParts: N ------=_Part_348126_1685684535.1677278715318 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline On Feb 23, 2023 at 11:15 AM, Chris Watson wrote: [Sorry miroslav, I hit send without checking the To: this was meant to be p= ublic]=C2=A0 I=E2=80=99m a bit late, but I mentioned this to someone on this thread priv= ately, I=E2=80=99m curious why =E2=80=98spiped=E2=80=99 hasn=E2=80=99t been= mentioned in this thread. I=E2=80=99ve seen everything from VPN=E2=80=99s = to nc. VPNs would be, imo, grossly unwarranted/massively overly complex/har= d to secure just to simply have a secure pipe for doing ZFS send|recv.=C2= =A0 Simply configuring an spiped PtP pipe between A and B seems the simplest, m= ost secure, performant option here. At least considering all the other opti= ons tossed out in this thread.=C2=A0 No one=E2=80=99s using spiped? O.o Thoughts?=C2=A0 Has anyone compared ssh to spiped regarding overhead and throughput in this= scenario? Chris On Wed, Feb 22, 2023 at 9:29 PM Sysadmin Lists wrote: On Feb 22, 2023 at 1:43 PM, Freddie Cash wrote: [Sorry for top part, GMail sucks for replies.] If this is a LAN or private WAN where you trust the network, piping the sen= d stream through netcat will remove ssh from the equation. That's what we switched to using once it became almost impossible to get th= e "none" cipher working with ssh on FreeBSD. We use ssh to connect to the remote server and enable a netcat listener on = port X, then pipe the send through netcat to the remote system on port X. T= hat way it's logged and uses ssh for authentication. We easily saturate gigabit links between our ZFS systems using netcat. Cheers, Freddie Typos due to smartphone keyboard. On Wed., Feb. 22, 2023, 1:31 p.m. Miroslav Lachman, <000.fbsd@quip.cz> wrot= e: On 22/02/2023 22:08, mike tancsa wrote: > On 2/22/2023 4:03 PM, Miroslav Lachman wrote: >> Interresting numbers. I think I am the only one who get best speed=20 >> with chacha20-poly1305@openssh.com >> >> >> It seems the speed of SSH is limited by single core performance which=20 >> is very poor on this machine (Intel(R) Pentium(R) Dual=C2=A0 CPU E2160).= =20 >> Even if CPU has 50% idle, ssh runs on 99.8% of single core. >=20 > The CPU I have has > aesni0: on motherboard >=20 > which probably helps. That explains it aesni0: No AES or SHA support. >> I know there were some HPN patches to ssh, beside that is there any=20 >> option I can try to use less CPU? >> >> I will play with cpuset to pin ssh on one core and everything else on=20 >> the other core. >=20 > It looks like you are running into a CPU bottleneck TBH Yes. Pinning on cores with cpuset helps a bit (about +3MiB/s) but=20 without some tweaks on ssh I will not gain more speed :( Thank you for your help! Miroslav Lachman You could pipe the stream through an encrypting program before piping to netcat, then decrypt on the recieving end. $ zfs send | crypt | netcat ipaddr 2222 $ netcat -vl 2222 | crypt | zfs recv I don't know if zfs can handle that, but worth a try. $ man crypt =C2=A0 =C2=A0 The enigma utility, also known as crypt is a very simple encr= yption =C2=A0 =C2=A0 =C2=A0program, working on a =E2=80=9Csecret-key=E2=80=9D basi= s.=C2=A0 It operates as a filter, i.e., =C2=A0 =C2=A0 =C2=A0it encrypts or decrypts a stream of data from standard = input, and writes =C2=A0 =C2=A0 =C2=A0the result to standard output.=C2=A0 Since its operatio= n is fully symmetrical, =C2=A0 =C2=A0 =C2=A0feeding the encrypted data stream again through the eng= ine (using the =C2=A0 =C2=A0 =C2=A0same secret key) will decrypt it. -- Sent with https://mailfence.com Secure and private email I've used it before, but forgot about it. But it's not part of base, and th= ere are tools in base which together perform a similar task, so that probably explains why many people haven't heard about it or forgot they had. Most everyone has at some point needed to transfer a couple files to a loca= l machine with a LAN connection but borked authentication services. In steps nc and optionally crypt or openssl to encrypt the data. Simple. -- Sent with https://mailfence.com Secure and private email ------=_Part_348126_1685684535.1677278715318 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline
=
On Feb 23, 2023 at 11:15 AM, Chris Watson <bsdunix44@gmail.com> = wrote:
">
[Sorry miroslav, I hit send without checking the To: this was mea= nt to be public] 

I=E2=80=99m a bit late, but I mentioned this to = someone on this thread privately, I=E2=80=99m curious why =E2=80=98spiped= =E2=80=99 hasn=E2=80=99t been mentioned in this thread. I=E2=80=99ve seen e= verything from VPN=E2=80=99s to nc. VPNs would be, imo, grossly unwarranted= /massively overly complex/hard to secure just to simply have a secure pipe = for doing ZFS send|recv. 

Simply configuring an spiped PtP pipe between A and B s= eems the simplest, most secure, performant option here. At least considerin= g all the other options tossed out in this thread. 

No one=E2=80=99s using spiped= ? O.o

Thoug= hts? 

= Has anyone compared ssh to spiped regarding overhead and throughput in this= scenario?

Chris

On Wed, Feb 22, 2023 at 9:29 PM Sysadmin Lists <sysadmin.lists@mailfence.com> wrot= e:

On Feb 22, 2023 at 1:43 PM, Freddie Cash <fjwcash@gmail.com> wrote:
[Sorry for top part, GMail suc= ks for replies.]

If this is a LAN or private WAN where you trust the network, pi= ping the send stream through netcat will remove ssh from the equation.

That's wh= at we switched to using once it became almost impossible to get the "none" = cipher working with ssh on FreeBSD.

We use ssh to connect to the remote se= rver and enable a netcat listener on port X, then pipe the send through net= cat to the remote system on port X. That way it's logged and uses ssh for a= uthentication.

We easily saturate gigabit links between our ZFS systems us= ing netcat.



Cheers,
Freddie
=
Typos due to smartphone keyboard.

On Wed., Feb. 22, 2023, 1:31 p.m. Miro= slav Lachman, <000.fbsd@quip.cz> wro= te:
On 22/02/2023 22:08, mike t= ancsa wrote:
> On 2/22/2023 4:03 PM, Miroslav Lachman wrote:
>> Interresting numbers. I think I am the only one who get best speed=
>> with = chacha20-poly1305@openssh.com
>>
>>
>> It seems the speed of SSH is limited by single core performance wh= ich
>> is very poor on this machine (Intel(R) Pentium(R) Dual  CPU E= 2160).
>> Even if CPU has 50% idle, ssh runs on 99.8% of single core.
>
> The CPU I have has
> aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS> on motherboard=
>
> which probably helps.

That explains it
aesni0: No AES or SHA support.

>> I know there were some HPN patches to ssh, beside that is there an= y
>> option I can try to use less CPU?
>>
>> I will play with cpuset to pin ssh on one core and everything else= on
>> the other core.
>
> It looks like you are running into a CPU bottleneck TBH

Yes. Pinning on cores with cpuset helps a bit (about +3MiB/s) but
without some tweaks on ssh I will not gain more speed :(

Thank you for your help!

Miroslav Lachman



You could = pipe the stream through an encrypting program before piping to
netcat, then decrypt on the= recieving end.

$ zfs sen= d | crypt | netcat ipaddr 2222
$ netcat -vl 2222 | crypt | zfs recv

I don't know if zfs can handle that, but worth = a try.
$ man crypt<= /div>
    The enigma utility, a= lso known as crypt is a very simple encryption
     program, working on a = =E2=80=9Csecret-key=E2=80=9D basis.  It operates as a filter, i.e.,
    &nb= sp;it encrypts or decrypts a stream of data from standard input, and writes=
    =  the result to standard output.  Since its operation is fully sym= metrical,
 = ;    feeding the encrypted data stream again through the engine (= using the
 = ;    same secret key) will decrypt it.


--=20 Sent with https://mailf= ence.com =20 Secure and private email

I've used it before, but forgot about it. But it's not part of base,= and there
are tools in base which together perform a similar tas= k, so that probably
explains why many people haven't heard about = it or forgot they had.

Most everyone has at some p= oint needed to transfer a couple files to a local
machine with a = LAN connection but borked authentication services. In steps
nc an= d optionally crypt or openssl to encrypt the data. Simple.


--=20 Sent with https://mailfence.com =20 Secure and private email ------=_Part_348126_1685684535.1677278715318--