From nobody Sat Dec 23 21:19:28 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SyH9p27Qkz551XD for ; Sat, 23 Dec 2023 21:19:42 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-ej1-x62d.google.com (mail-ej1-x62d.google.com [IPv6:2a00:1450:4864:20::62d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SyH9n42pyz4Q8j for ; Sat, 23 Dec 2023 21:19:41 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20230601.gappssmtp.com header.s=20230601 header.b=fXBBumwa; spf=none (mx1.freebsd.org: domain of wlosh@bsdimp.com has no SPF policy when checking 2a00:1450:4864:20::62d) smtp.mailfrom=wlosh@bsdimp.com; dmarc=none Received: by mail-ej1-x62d.google.com with SMTP id a640c23a62f3a-a234dc0984fso305453766b.0 for ; Sat, 23 Dec 2023 13:19:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20230601.gappssmtp.com; s=20230601; t=1703366380; x=1703971180; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=YVvYFyS/TB5JqAVd5J29h47ymdD/CyXCkj3tXa/ec9Q=; b=fXBBumwabSHgmV5baRAMPkl835iPNjQ3T8okPx/tXLYuClKpnEoQUxjnOTSt+dWHOK CvS0rwCtz9LikwYC8YnDyWeZ9W3d0SeCICoeGLOgignOQ5HiPpPM17coOU4kzVzEZvty dwg/hymvKdQx3EE0VVwE6lQfztKEzbpywQ4jyPglmgNzFJ5zu/aupxS3lEfq1RGu0Zqo I0JWg+HFJ4v6MeC0y8FdDZx4Cd6ExBEvdL7WrOrgZsIE1vXwncVpZu3Bk9jdQiqTpR2u dzakpPOQ6HMnJon0okhFRPcxNM2RNhE7VdXWMZvOmgg7L0KHLYtPMMZ+COPIlVOGbFGZ 2oJQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703366380; x=1703971180; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=YVvYFyS/TB5JqAVd5J29h47ymdD/CyXCkj3tXa/ec9Q=; b=M1BUl3vgALBSVZSDJUvjYUt5B9SPvnR40MnRUcxnveg3Jnu31tcbLQ2Q6IjhpxDxdo rmt1cJdcGIe+xhnJXRFdBE4EnV3PgVf5JHdSI3cSx52jW9LLN/RKgH+Lr3iVvVK143E7 sf5i1DYW9uH1Jx8dW33dt19winISpXlfAIKWr0Lh0uIcweHhbkVn/cQY+u5Pdh5mZzEd brdT53OE7cL1UVbMpMdeo5+ePI4g84Ai2kLZZLGxPMbyTJVsfs90hzRESUgeUFiKdpLZ fxKn9qUq+9waO6D/IvHgUaXp08Nsi15I5qisiqHoQyNFTyP0tGPexEv3cDd0KbBjhES/ izoQ== X-Gm-Message-State: AOJu0Yxpcb6qm7tjig0muQLBoWSIriSLuiL4rpradE+jyOsIm6AgqNor C306knAflcqN700vFc8DpDnvIGnTvK5cIrX/5nJYtBucTfTCZjTtJc26Av/ag7U= X-Google-Smtp-Source: AGHT+IEscmjia9edp+BW3RM5Nl699dYmTBLsX0RBdP3Uk//muMbVdx1Bl8sD8toQtrSIgCBnmAYI0cHn6aFvdDH8aIA= X-Received: by 2002:a17:906:6954:b0:a26:97dc:67d9 with SMTP id c20-20020a170906695400b00a2697dc67d9mr1732723ejs.68.1703366379679; Sat, 23 Dec 2023 13:19:39 -0800 (PST) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 References: <2dcc21bb-f446-41fa-871b-2c37cdf9afe0@app.fastmail.com> In-Reply-To: <2dcc21bb-f446-41fa-871b-2c37cdf9afe0@app.fastmail.com> From: Warner Losh Date: Sat, 23 Dec 2023 14:19:28 -0700 Message-ID: Subject: Re: measuring swap partition speed To: void Cc: freebsd-fs Content-Type: multipart/alternative; boundary="000000000000fbc1c2060d33e351" X-Spamd-Result: default: False [-3.00 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20230601.gappssmtp.com:s=20230601]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCVD_COUNT_ONE(0.00)[1]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; FREEMAIL_TO(0.00)[f-m.fm]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; R_SPF_NA(0.00)[no SPF record]; RCVD_TLS_LAST(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::62d:from]; DKIM_TRACE(0.00)[bsdimp-com.20230601.gappssmtp.com:+]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; DMARC_NA(0.00)[bsdimp.com]; TO_DN_ALL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com] X-Rspamd-Queue-Id: 4SyH9n42pyz4Q8j X-Spamd-Bar: -- --000000000000fbc1c2060d33e351 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Thu, Dec 21, 2023 at 5:20=E2=80=AFPM void wrote: > Hi Mark & Warner, > > I tried using fio with Warner's suggestions as a template, and > pasted the results in the latter part of this message. > > On Thu, 21 Dec 2023, at 21:03, Mark Millard wrote: > > > # sysctl vm.stats.vm.v_page_size > > vm.stats.vm.v_page_size: 4096 > > Here, this setting is: > # sysctl vm.stats.vm.v_page_size > vm.stats.vm.v_page_size: 4096 > > > # sysctl vm.phys_pager_cluster > > vm.phys_pager_cluster: 1024 > > It is 1024 here, too. > > Only the one usb3 port is occupied. > > fio output follows. No idea yet if the settings used are suitable for thi= s > context > and would welcome suggestions. I think --filename can be a device? Output > suggests > it used hw.ncpu instead of --numjobs=3D8 > > ~~~~~~~ > # fio --name=3Drandread --ioengine=3Dposixaio --rw=3Drandread --direct=3D= 1 --bs=3D8k > --refill_buffers --norandommap --randrepeat=3D0 --iodepth=3D32 --numjobs= =3D8 > --runtime=3D60 --group_reporting --thread --size=3D2048M > randread: (g=3D0): rw=3Drandread, bs=3D(R) 8192B-8192B, (W) 8192B-8192B, = (T) > 8192B-8192B, ioengine=3Dposixaio, iodepth=3D32 > .. > fio-3.36 > Starting 8 threads > randread: Laying out IO file (1 file / 2048MiB) > randread: Laying out IO file (1 file / 2048MiB) > randread: Laying out IO file (1 file / 2048MiB) > randread: Laying out IO file (1 file / 2048MiB) > randread: Laying out IO file (1 file / 2048MiB) > randread: Laying out IO file (1 file / 2048MiB) > randread: Laying out IO file (1 file / 2048MiB) > randread: Laying out IO file (1 file / 2048MiB) > Jobs: 4 (f=3D4): [_(1),r(2),_(2),r(1),_(1),r(1)][21.2%][r=3D288KiB/s][r= =3D36 > IOPS][eta 04m:05s] > randread: (groupid=3D0, jobs=3D8): err=3D 0: pid=3D135125: Thu Dec 21 16:= 43:00 2023 > read: IOPS=3D43, BW=3D351KiB/s (359kB/s)(22.6MiB/65986msec) > slat (nsec): min=3D889, max=3D1948.6k, avg=3D4172.45, stdev=3D36488.0= 7 > clat (msec): min=3D1108, max=3D11660, avg=3D5644.23, stdev=3D1282.12 > lat (msec): min=3D1108, max=3D11660, avg=3D5644.24, stdev=3D1282.12 > clat percentiles (msec): > | 1.00th=3D[ 1183], 5.00th=3D[ 3171], 10.00th=3D[ 4933], 20.00th= =3D[ 5269], > | 30.00th=3D[ 5470], 40.00th=3D[ 5604], 50.00th=3D[ 5738], 60.00th= =3D[ 5873], > | 70.00th=3D[ 5940], 80.00th=3D[ 6074], 90.00th=3D[ 6342], 95.00th= =3D[ 6812], > | 99.00th=3D[10671], 99.50th=3D[10939], 99.90th=3D[11610], 99.95th= =3D[11610], > | 99.99th=3D[11610] > bw ( KiB/s): min=3D 208, max=3D 3760, per=3D100.00%, avg=3D1535.05, > stdev=3D128.23, samples=3D245 > iops : min=3D 26, max=3D 470, avg=3D191.88, stdev=3D16.03, s= amples=3D245 > lat (msec) : 2000=3D3.25%, >=3D2000=3D96.75% > cpu : usr=3D0.00%, sys=3D0.12%, ctx=3D22712, majf=3D0, minf=3D= 0 > IO depths : 1=3D0.3%, 2=3D0.6%, 4=3D1.1%, 8=3D4.9%, 16=3D69.6%, 32= =3D23.6%, > >=3D64=3D0.0% > submit : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%, 32=3D0.0%, 64= =3D0.0%, > >=3D64=3D0.0% > complete : 0=3D0.0%, 4=3D94.7%, 8=3D2.9%, 16=3D1.5%, 32=3D0.9%, 64= =3D0.0%, > >=3D64=3D0.0% > issued rwts: total=3D2895,0,0,0 short=3D0,0,0,0 dropped=3D0,0,0,0 > latency : target=3D0, window=3D0, percentile=3D100.00%, depth=3D32 > > Run status group 0 (all jobs): > READ: bw=3D351KiB/s (359kB/s), 351KiB/s-351KiB/s (359kB/s-359kB/s), > io=3D22.6MiB (23.7MB), run=3D65986-65986msec > # > 5s+ of latency on the average, max latency of 12s! Woof. No wonder you hate life. Warner --000000000000fbc1c2060d33e351 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Thu, Dec 21, 2023 at 5:20=E2=80=AF= PM void <void@f-m.fm> wrote:
Hi Mark & Warner,=

I tried using fio with Warner's suggestions as a template, and
pasted the results in the latter part of this message.

On Thu, 21 Dec 2023, at 21:03, Mark Millard wrote:

> # sysctl vm.stats.vm.v_page_size
> vm.stats.vm.v_page_size: 4096

Here, this setting is:
# sysctl vm.stats.vm.v_page_size
vm.stats.vm.v_page_size: 4096

> # sysctl vm.phys_pager_cluster
> vm.phys_pager_cluster: 1024

It is 1024 here, too.

Only the one usb3 port is occupied.

fio output follows. No idea yet if the settings used are suitable for this = context
and would welcome suggestions. I think --filename can be a device? Output s= uggests
it used hw.ncpu instead of --numjobs=3D8

~~~~~~~
# fio --name=3Drandread --ioengine=3Dposixaio --rw=3Drandread --direct=3D1 = --bs=3D8k --refill_buffers --norandommap --randrepeat=3D0 --iodepth=3D32 --= numjobs=3D8 --runtime=3D60 --group_reporting --thread --size=3D2048M
randread: (g=3D0): rw=3Drandread, bs=3D(R) 8192B-8192B, (W) 8192B-8192B, (T= ) 8192B-8192B, ioengine=3Dposixaio, iodepth=3D32
..
fio-3.36
Starting 8 threads
randread: Laying out IO file (1 file / 2048MiB)
randread: Laying out IO file (1 file / 2048MiB)
randread: Laying out IO file (1 file / 2048MiB)
randread: Laying out IO file (1 file / 2048MiB)
randread: Laying out IO file (1 file / 2048MiB)
randread: Laying out IO file (1 file / 2048MiB)
randread: Laying out IO file (1 file / 2048MiB)
randread: Laying out IO file (1 file / 2048MiB)
Jobs: 4 (f=3D4): [_(1),r(2),_(2),r(1),_(1),r(1)][21.2%][r=3D288KiB/s][r=3D3= 6 IOPS][eta 04m:05s]
randread: (groupid=3D0, jobs=3D8): err=3D 0: pid=3D135125: Thu Dec 21 16:43= :00 2023
=C2=A0 read: IOPS=3D43, BW=3D351KiB/s (359kB/s)(22.6MiB/65986msec)
=C2=A0 =C2=A0 slat (nsec): min=3D889, max=3D1948.6k, avg=3D4172.45, stdev= =3D36488.07
=C2=A0 =C2=A0 clat (msec): min=3D1108, max=3D11660, avg=3D5644.23, stdev=3D= 1282.12
=C2=A0 =C2=A0 =C2=A0lat (msec): min=3D1108, max=3D11660, avg=3D5644.24, std= ev=3D1282.12
=C2=A0 =C2=A0 clat percentiles (msec):
=C2=A0 =C2=A0 =C2=A0|=C2=A0 1.00th=3D[ 1183],=C2=A0 5.00th=3D[ 3171], 10.00= th=3D[ 4933], 20.00th=3D[ 5269],
=C2=A0 =C2=A0 =C2=A0| 30.00th=3D[ 5470], 40.00th=3D[ 5604], 50.00th=3D[ 573= 8], 60.00th=3D[ 5873],
=C2=A0 =C2=A0 =C2=A0| 70.00th=3D[ 5940], 80.00th=3D[ 6074], 90.00th=3D[ 634= 2], 95.00th=3D[ 6812],
=C2=A0 =C2=A0 =C2=A0| 99.00th=3D[10671], 99.50th=3D[10939], 99.90th=3D[1161= 0], 99.95th=3D[11610],
=C2=A0 =C2=A0 =C2=A0| 99.99th=3D[11610]
=C2=A0 =C2=A0bw (=C2=A0 KiB/s): min=3D=C2=A0 208, max=3D 3760, per=3D100.00= %, avg=3D1535.05, stdev=3D128.23, samples=3D245
=C2=A0 =C2=A0iops=C2=A0 =C2=A0 =C2=A0 =C2=A0 : min=3D=C2=A0 =C2=A026, max= =3D=C2=A0 470, avg=3D191.88, stdev=3D16.03, samples=3D245
=C2=A0 lat (msec)=C2=A0 =C2=A0: 2000=3D3.25%, >=3D2000=3D96.75%
=C2=A0 cpu=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : usr=3D0.00%, sys=3D0.12%, ct= x=3D22712, majf=3D0, minf=3D0
=C2=A0 IO depths=C2=A0 =C2=A0 : 1=3D0.3%, 2=3D0.6%, 4=3D1.1%, 8=3D4.9%, 16= =3D69.6%, 32=3D23.6%, >=3D64=3D0.0%
=C2=A0 =C2=A0 =C2=A0submit=C2=A0 =C2=A0 : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 1= 6=3D0.0%, 32=3D0.0%, 64=3D0.0%, >=3D64=3D0.0%
=C2=A0 =C2=A0 =C2=A0complete=C2=A0 : 0=3D0.0%, 4=3D94.7%, 8=3D2.9%, 16=3D1.= 5%, 32=3D0.9%, 64=3D0.0%, >=3D64=3D0.0%
=C2=A0 =C2=A0 =C2=A0issued rwts: total=3D2895,0,0,0 short=3D0,0,0,0 dropped= =3D0,0,0,0
=C2=A0 =C2=A0 =C2=A0latency=C2=A0 =C2=A0: target=3D0, window=3D0, percentil= e=3D100.00%, depth=3D32

Run status group 0 (all jobs):
=C2=A0 =C2=A0READ: bw=3D351KiB/s (359kB/s), 351KiB/s-351KiB/s (359kB/s-359k= B/s), io=3D22.6MiB (23.7MB), run=3D65986-65986msec
#

5s+ of latency on the average, max la= tency of 12s!=C2=A0 Woof. No wonder you hate life.

Warner
--000000000000fbc1c2060d33e351--