From nobody Fri Dec 22 09:59:26 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SxN7Q36mZz552SV for ; Fri, 22 Dec 2023 09:59:30 +0000 (UTC) (envelope-from SRS0=ck8o=IB=klop.ws=ronald-lists@realworks.nl) Received: from smtp-relay-int.realworks.nl (smtp-relay-int.realworks.nl [194.109.157.24]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4SxN7P1dsxz4QPN for ; Fri, 22 Dec 2023 09:59:29 +0000 (UTC) (envelope-from SRS0=ck8o=IB=klop.ws=ronald-lists@realworks.nl) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=klop.ws header.s=rw2 header.b=SD1t+Q6G; spf=pass (mx1.freebsd.org: domain of "SRS0=ck8o=IB=klop.ws=ronald-lists@realworks.nl" designates 194.109.157.24 as permitted sender) smtp.mailfrom="SRS0=ck8o=IB=klop.ws=ronald-lists@realworks.nl"; dmarc=pass (policy=quarantine) header.from=klop.ws Date: Fri, 22 Dec 2023 10:59:26 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=klop.ws; s=rw2; t=1703239166; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=1+61QWft+NhOOAqGeRidYRz8+Zo04FPInMd7U/tlm8Q=; b=SD1t+Q6G4m2KSRP5zFtFtW12Tuq4a4o7JhQNiAaSC813Bi1yImZYemsIHcGSv1OjN8p384 tEgChl6eUUu1yI1DeSJhvWUIlXfu3yydrhNfpZ/lCxUgm+PmnUcCbm+zZNVIRJP47PVFAz VMv7EMUjYra7Kc3iD47h/sMIcYmlNsjvU26uI47cApMzRjC+XGG2a6/kzM5E0AmgLjd58m 2st+0sRHmPj3em+9d8cUjy++eukgzpg647OAd6QsBlO1TrPsc0anfKyh8Kw1uCk0lRlr93 RDJXf2HXFGYTSvw/FXXIF2dogaL6+iQytYeYMonwPyiXdfJ52iT4QNbMoFTZrA== From: Ronald Klop To: freebsd-fs , void Message-ID: <478987228.2760.1703239166835@localhost> In-Reply-To: <1137397689.3080.1703237862787@localhost> References: <1439231787.9333.1703184981914@localhost> <1dbf8804-f1ac-4578-a538-889744d7de9a@app.fastmail.com> <1137397689.3080.1703237862787@localhost> Subject: Re: measuring swap partition speed List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_2759_327864946.1703239166820" X-Mailer: Realworks (683.54) Importance: Normal X-Priority: 3 (Normal) X-Spamd-Result: default: False [-3.20 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; DMARC_POLICY_ALLOW(-0.50)[klop.ws,quarantine]; MID_RHS_NOT_FQDN(0.50)[]; FORGED_SENDER(0.30)[ronald-lists@klop.ws,SRS0=ck8o=IB=klop.ws=ronald-lists@realworks.nl]; R_DKIM_ALLOW(-0.20)[klop.ws:s=rw2]; R_SPF_ALLOW(-0.20)[+ip4:194.109.157.0/24]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; FREEMAIL_TO(0.00)[freebsd.org,f-m.fm]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_COUNT_ZERO(0.00)[0]; ARC_NA(0.00)[]; DKIM_TRACE(0.00)[klop.ws:+]; RCPT_COUNT_TWO(0.00)[2]; FROM_HAS_DN(0.00)[]; HAS_X_PRIO_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; ASN(0.00)[asn:3265, ipnet:194.109.0.0/16, country:NL]; TO_DN_ALL(0.00)[]; FROM_NEQ_ENVFROM(0.00)[ronald-lists@klop.ws,SRS0=ck8o=IB=klop.ws=ronald-lists@realworks.nl] X-Rspamd-Queue-Id: 4SxN7P1dsxz4QPN X-Spamd-Bar: --- ------=_Part_2759_327864946.1703239166820 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Van: Ronald Klop Datum: vrijdag, 22 december 2023 10:37 Aan: void CC: freebsd-fs Onderwerp: Re: measuring swap partition speed > > > Van: void > Datum: vrijdag, 22 december 2023 01:05 > Aan: freebsd-fs > Onderwerp: Re: measuring swap partition speed >> >> On Thu, 21 Dec 2023, at 18:56, Ronald Klop wrote: >> > A bit weird that a simple dd is so slow. >> > >> > Just a quick thought. >> > Are your partitions aligned properly? >> >> no idea. How would I check? The system when installed, >> the auto-zfs option was selected, block size set to 4k. The swap >> was changed from the default 2gb to 12gb. geli encryption is >> active for the zfs filesystem but not for swap. It's only the swap >> that shows performance issues. >> >> > Is other IO going on on the same time? >> >> not really, the tests were done with cron jobs deactivated, almost completely idle, >> swap was not being used at all. >> >> > What does gstat say about %util, queue length and all the other stats >> > while running the dd? Or "iostat -x -d 1". >> >> I'll try that next, thanks >> >> > Could you try if another disk has the same issues? >> >> no spare or equivalent disk for this arch. >> >> > How is your disk connected? USB-to-SATA-adapter? Any output of dmesg? >> >> Regular usb3 connection. >> >> da0: 400.000MB/s transfers >> da0: 953869MB (1953525168 512 byte sectors) >> da0: quirks=0x2 >> da0: Delete methods: >> -- >> >> >> >> > > > Can you provide more concrete information? Like: > > # usbconfig list > ugen0.1: <(0x1106) XHCI root HUB> at usbus0, cfg=0 md=HOST spd=SUPER (5.0Gbps) pwr=SAVE (0mA) > ugen0.2: at usbus0, cfg=0 md=HOST spd=HIGH (480Mbps) pwr=SAVE (100mA) > ugen0.3: at usbus0, cfg=0 md=HOST spd=FULL (12Mbps) pwr=ON (100mA) > ugen0.4: at usbus0, cfg=0 md=HOST spd=SUPER (5.0Gbps) pwr=ON (224mA) > ugen0.5: at usbus0, cfg=0 md=HOST spd=SUPER (5.0Gbps) pwr=ON (224mA) > > And the output of "devinfo -v". That will give people much more information to work with instead of "anecdotal evidence" about your hardware being usb3. > And the complete output of dmesg instead of only the "da0:" lines. There is more information in there which can be valuable for people to help you. > > About the alignment. You wrote in your first mail: > "# gpart show > > => 40 1953525088 da0 GPT (932G) > 40 532480 1 efi (260M) > 532520 2008 - free - (1.0M) > 534528 4194304 2 freebsd-swap (2.0G) > 4728832 4194304 4 freebsd-swap (2.0G) > 8923136 4194304 5 freebsd-swap (2.0G) > 13117440 4194304 6 freebsd-swap (2.0G) > 17311744 4194304 7 freebsd-swap (2.0G) > 21506048 4194304 8 freebsd-swap (2.0G) > 25700352 1927823360 3 freebsd-zfs (920G) > 1953523712 1416 - free - (708K) > " > > Your dd tests were on da0p4 which starts on block 4728832. > Assuming "gpart list" shows "Sectorsize: 512". > (4728832 * 512) mod 4096 = 0, so the partitiion is aligned on a 4KB page size. And the calculation is also 0 for 8KB (is this the page size of arm64?) which is good. > AFAIK: this is properly aligned such that the hardware does not need to double write or read-before-write when paging out. > > Up until now I haven't seen anything which would explain the slow dd speed on of=/dev/da0p4 vs the quick dd on of=/mnt/test8k.bin. > > Regards, > Ronald. > Just did the same test on my RPI4 running FreeBSD 15.0-CURRENT of Dec 14. Used an unused partition on spinning disk. My used swap is on an ssd nowadays. [root@rpi4 ~]# dd if=/dev/urandom of=/dev/da1p2 bs=8k count=25000 conv=sync status=progress 193200128 bytes (193 MB, 184 MiB) transferred 10.024s, 19 MB/s 25000+0 records in 25000+0 records out 204800000 bytes transferred in 10.592984 secs (19333551 bytes/sec) NB: using bs=128k makes it go to 69MB/s. # gpart show => 40 1953458096 da1 GPT (931G) 40 102400 1 efi (50M) 102440 8388608 2 freebsd-swap (4.0G) 8491048 1944967088 3 freebsd-zfs (927G) ugen0.5: at usbus0 umass1 on uhub0 umass1: on usbus0 da1 at umass-sim1 bus 1 scbus1 target 0 lun 0 da1: Fixed Direct Access SPC-4 SCSI device da1: Serial Number 575843314538354834414536 da1: 400.000MB/s transfers da1: 953837MB (1953458176 512 byte sectors) da1: quirks=0x2 # usbconfig list ugen0.1: <(0x1106) XHCI root HUB> at usbus0, cfg=0 md=HOST spd=SUPER (5.0Gbps) pwr=SAVE (0mA) ugen0.2: at usbus0, cfg=0 md=HOST spd=HIGH (480Mbps) pwr=SAVE (100mA) ugen0.3: at usbus0, cfg=0 md=HOST spd=FULL (12Mbps) pwr=ON (100mA) ugen0.4: at usbus0, cfg=0 md=HOST spd=SUPER (5.0Gbps) pwr=ON (224mA) ugen0.5: at usbus0, cfg=0 md=HOST spd=SUPER (5.0Gbps) pwr=ON (224mA) Found some more things to check: Do these settings have something non-default? # sysctl kern.geom.disk # sysctl kern.cam And your disk prints this in dmesg. How does that influence things? I haven't seen this before anywhere. da0: Delete methods: Because of this I'm particularly interested in: sysctl kern.cam.da.0.delete_method and sysctl kern.cam.da.0.trim_count I might be totally of in my thinking here, but you never know with such unpredictable problems. Regards, Ronald. ------=_Part_2759_327864946.1703239166820 Content-Type: text/html; charset=us-ascii Content-Transfer-Encoding: 7bit

Van: Ronald Klop <ronald-lists@klop.ws>
Datum: vrijdag, 22 december 2023 10:37
Aan: void <void@f-m.fm>
CC: freebsd-fs <freebsd-fs@freebsd.org>
Onderwerp: Re: measuring swap partition speed

 

Van: void <void@f-m.fm>
Datum: vrijdag, 22 december 2023 01:05
Aan: freebsd-fs <freebsd-fs@freebsd.org>
Onderwerp: Re: measuring swap partition speed

On Thu, 21 Dec 2023, at 18:56, Ronald Klop wrote:
> A bit weird that a simple dd is so slow.
>
> Just a quick thought.
> Are your partitions aligned properly?

no idea. How would I check? The system when installed,
the auto-zfs option was selected, block size set to 4k. The swap
was changed from the default 2gb to 12gb. geli encryption is
active for the zfs filesystem but not for swap. It's only the swap
that shows performance issues.

> Is other IO going on on the same time?

not really, the tests were done with cron jobs deactivated, almost completely idle,
swap was not being used at all.

> What does gstat say about %util, queue length and all the other stats
> while running the dd? Or "iostat -x -d 1".

I'll try that next, thanks

> Could you try if another disk has the same issues?

no spare or equivalent disk for this arch.

> How is your disk connected? USB-to-SATA-adapter? Any output of dmesg?

Regular usb3 connection.

da0: 400.000MB/s transfers
da0: 953869MB (1953525168 512 byte sectors)
da0: quirks=0x2<NO_6_BYTE>
da0: Delete methods: <NONE(*),ZERO>
-- 
 


Can you provide more concrete information? Like:

# usbconfig list
ugen0.1: <(0x1106) XHCI root HUB> at usbus0, cfg=0 md=HOST spd=SUPER (5.0Gbps) pwr=SAVE (0mA)
ugen0.2: <vendor 0x2109 USB2.0 Hub> at usbus0, cfg=0 md=HOST spd=HIGH (480Mbps) pwr=SAVE (100mA)
ugen0.3: <Silicon Labs CP2102 USB to UART Bridge Controller> at usbus0, cfg=0 md=HOST spd=FULL (12Mbps) pwr=ON (100mA)
ugen0.4: <USB 3.0 Device USB 3.0 Device> at usbus0, cfg=0 md=HOST spd=SUPER (5.0Gbps) pwr=ON (224mA)
ugen0.5: <Western Digital Elements 25A2> at usbus0, cfg=0 md=HOST spd=SUPER (5.0Gbps) pwr=ON (224mA)

And the output of "devinfo -v". That will give people much more information to work with instead of "anecdotal evidence" about your hardware being usb3.
And the complete output of dmesg instead of only the "da0:" lines. There is more information in there which can be valuable for people to help you.

About the alignment. You wrote in your first mail:
"# gpart show

=>        40  1953525088  da0  GPT  (932G)
           40      532480    1  efi  (260M)
       532520        2008       - free -  (1.0M)
       534528     4194304    2  freebsd-swap  (2.0G)
      4728832     4194304    4  freebsd-swap  (2.0G)
      8923136     4194304    5  freebsd-swap  (2.0G)
     13117440     4194304    6  freebsd-swap  (2.0G)
     17311744     4194304    7  freebsd-swap  (2.0G)
     21506048     4194304    8  freebsd-swap  (2.0G)
     25700352  1927823360    3  freebsd-zfs  (920G)
   1953523712        1416       - free -  (708K)
"

Your dd tests were on da0p4 which starts on block 4728832.
Assuming "gpart list" shows "Sectorsize: 512".
(4728832 * 512) mod 4096 = 0, so the partitiion is aligned on a 4KB page size. And the calculation is also 0 for 8KB (is this the page size of arm64?) which is good.
AFAIK: this is properly aligned such that the hardware does not need to double write or read-before-write when paging out.

Up until now I haven't seen anything which would explain the slow dd speed on of=/dev/da0p4 vs the quick dd on of=/mnt/test8k.bin.

Regards,
Ronald.
 

Just did the same test on my RPI4 running FreeBSD 15.0-CURRENT of Dec 14.
Used an unused partition on spinning disk. My used swap is on an ssd nowadays.

[root@rpi4 ~]# dd if=/dev/urandom of=/dev/da1p2 bs=8k count=25000 conv=sync status=progress
  193200128 bytes (193 MB, 184 MiB) transferred 10.024s, 19 MB/s
25000+0 records in
25000+0 records out
204800000 bytes transferred in 10.592984 secs (19333551 bytes/sec)

NB: using bs=128k makes it go to 69MB/s.

# gpart show
=>        40  1953458096  da1  GPT  (931G)
          40      102400    1  efi  (50M)
      102440     8388608    2  freebsd-swap  (4.0G)
     8491048  1944967088    3  freebsd-zfs  (927G)

ugen0.5: <Western Digital Elements 25A2> at usbus0
umass1 on uhub0
umass1: <Western Digital Elements 25A2, class 0/0, rev 3.00/10.04, addr 4> on usbus0
da1 at umass-sim1 bus 1 scbus1 target 0 lun 0
da1: <WD Elements 25A2 1004> Fixed Direct Access SPC-4 SCSI device
da1: Serial Number 575843314538354834414536
da1: 400.000MB/s transfers
da1: 953837MB (1953458176 512 byte sectors)
da1: quirks=0x2<NO_6_BYTE>

# usbconfig list
ugen0.1: <(0x1106) XHCI root HUB> at usbus0, cfg=0 md=HOST spd=SUPER (5.0Gbps) pwr=SAVE (0mA)
ugen0.2: <vendor 0x2109 USB2.0 Hub> at usbus0, cfg=0 md=HOST spd=HIGH (480Mbps) pwr=SAVE (100mA)
ugen0.3: <Silicon Labs CP2102 USB to UART Bridge Controller> at usbus0, cfg=0 md=HOST spd=FULL (12Mbps) pwr=ON (100mA)
ugen0.4: <USB 3.0 Device USB 3.0 Device> at usbus0, cfg=0 md=HOST spd=SUPER (5.0Gbps) pwr=ON (224mA)
ugen0.5: <Western Digital Elements 25A2> at usbus0, cfg=0 md=HOST spd=SUPER (5.0Gbps) pwr=ON (224mA)


Found some more things to check:

Do these settings have something non-default?
# sysctl kern.geom.disk
# sysctl kern.cam

And your disk prints this in dmesg. How does that influence things? I haven't seen this before anywhere.
da0: Delete methods: <NONE(*),ZERO>

Because of this I'm particularly interested in:
sysctl kern.cam.da.0.delete_method
and
sysctl kern.cam.da.0.trim_count

I might be totally of in my thinking here, but you never know with such unpredictable problems.

Regards,
Ronald.
  ------=_Part_2759_327864946.1703239166820--