From nobody Thu Dec 28 12:24:14 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4T175n2ZPmz55lFj for ; Thu, 28 Dec 2023 12:26:05 +0000 (UTC) (envelope-from void@f-m.fm) Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com [66.111.4.28]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4T175m3cnmz4Nhp for ; Thu, 28 Dec 2023 12:26:04 +0000 (UTC) (envelope-from void@f-m.fm) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=f-m.fm header.s=fm2 header.b=azUPjVNC; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=2y+ae8WH; spf=pass (mx1.freebsd.org: domain of void@f-m.fm designates 66.111.4.28 as permitted sender) smtp.mailfrom=void@f-m.fm; dmarc=pass (policy=none) header.from=f-m.fm Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailout.nyi.internal (Postfix) with ESMTP id 91C4C5C009E for ; Thu, 28 Dec 2023 07:26:03 -0500 (EST) Received: from imap46 ([10.202.2.96]) by compute6.internal (MEProxy); Thu, 28 Dec 2023 07:26:03 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=f-m.fm; h=cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm2; t=1703766363; x=1703852763; bh=qZP/z5KvIg 2lFYyVgNPIPzfsFejRn8SNcYE6MxazKro=; b=azUPjVNC3RyXIXCHh230QEzojM C3VIkX70F4Fd0JBZUWq8xclXGtGIXE8TbMDWmoyUcQ5KnZrH9AqlDRhDGmoG6Xfq 7BQ2A41TRCt/d3kp++igrC4I/hCZz1YDTcnBcvQOpv0r1WW3yhcFi8/WSB/yPREd XpxWWgG/JMCzo48NgkjzupXdp/3sESyo/HSh9PcrgvE95+iXTCBHlw/HAkfxy1tm 28VVoGl8rGIQ6/PKgpf98TSouVW4gHLOr54TTYRJ+G0lL59l743SoFGjk/nsErn7 4rtBqgJAUc5NSaSkCRNy4q2RghAgmAw+SAGCXscy5ObSQzvIAMWYQhDag9ng== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; t=1703766363; x=1703852763; bh=qZP/z5KvIg2lFYyVgNPIPzfsFejR n8SNcYE6MxazKro=; b=2y+ae8WHRnC6R+Jkywr4ubxkVhHxn2o/MGOXJCmK7B0M NKabgOUVKl9RCkIxzQCGnFoWN6G1IBI+g5D2xz3qoe/auGFoZ9CWinv5PvJVpXvD Mi12NODfTEsR+QsI69VJHEb6XkPz1h8/e3MfwEhKlI8edLOXk5xdrYmUtAaRbrE5 GTbfMG3fZxe4zsx3yFBecPrEngnaAiaUI/fjQdhJwQhgrE6TY5VQXopbG8YlxRAE NnTlIsxxhyn5CX3LyWi9ZOYXhH6ew69P3NwnQ4rEa+PYDwm8fHtflYbvyUY/fhr/ b5YbNOk//E1RsjvrhGujB/6GF6g2F9jlU9jIAZr4fw== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedrvdefuddggeduucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpefofgggkfgjfhffhffvufgtsehttd ertderredtnecuhfhrohhmpehvohhiugcuoehvohhiugesfhdqmhdrfhhmqeenucggtffr rghtthgvrhhnpeeiieehuedtudeikeduieekvdffgefgfeevvdefheffgedutdehtdetvd fhveekhfenucffohhmrghinhepfhhrvggvsghsugdrohhrghenucevlhhushhtvghrufhi iigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehvohhiugesfhdqmhdrfhhm X-ME-Proxy: Feedback-ID: i2541463c:Fastmail Received: by mailuser.nyi.internal (Postfix, from userid 501) id 523722A2008B; Thu, 28 Dec 2023 07:26:03 -0500 (EST) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.9.0-alpha0-1364-ga51d5fd3b7-fm-20231219.001-ga51d5fd3 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Message-Id: In-Reply-To: References: <0D286183-ACD6-4818-8E27-E03D7F16D288@yahoo.com> Date: Thu, 28 Dec 2023 12:24:14 +0000 From: void To: freebsd-fs Subject: Re: measuring swap partition speed Content-Type: text/plain X-Spamd-Result: default: False [-5.11 / 15.00]; DWL_DNSWL_LOW(-1.00)[messagingengine.com:dkim]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.92)[-0.921]; DMARC_POLICY_ALLOW(-0.50)[f-m.fm,none]; R_DKIM_ALLOW(-0.20)[f-m.fm:s=fm2,messagingengine.com:s=fm2]; R_SPF_ALLOW(-0.20)[+ip4:66.111.4.28]; RCVD_IN_DNSWL_LOW(-0.10)[66.111.4.28:from]; RWL_MAILSPIKE_GOOD(-0.10)[66.111.4.28:from]; MIME_GOOD(-0.10)[text/plain]; XM_UA_NO_VERSION(0.01)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; ARC_NA(0.00)[]; ASN(0.00)[asn:19151, ipnet:66.111.4.0/24, country:US]; FREEMAIL_ENVFROM(0.00)[f-m.fm]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[f-m.fm]; RCVD_TLS_LAST(0.00)[]; TO_DN_ALL(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[f-m.fm:+,messagingengine.com:+]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org] X-Rspamd-Queue-Id: 4T175m3cnmz4Nhp X-Spamd-Bar: ----- Plugged the disk into usb3 interface on amd64 machine and ran the following [a] tests all at roughly the same time. The write test (2x git clone/pull) goes to a ufs partition. Writing 2x swap at the same time never got below 3mb/s tl;dr : 1. in the arm64 context, the extreme decline in swap performance happens if swap gets used, by anything (system) 2. in the [1] context, even when swap later becomes unused by the system, the performance problem persists. 3. the situation is recoverable only via swapoff then on. Then the swap partition can be written to in the tests at full speed. Writing subsequent swap tests also run at full speed. When the system swaps again, the problem returns. 4. the slowdown does not arise in an amd64 context beyond what one would expect 5. ufs or zfs makes no real difference it wouldn't matter to me if the write speed was 3mb, 16mb or 22mb/s. What causes problems is the sub-1mb/s speed (worst I've seen it is 300kb/s) and the non-recovery of that swap space when load eases off. I'm probably doing myself no favours using zfs in the arm64 context though. thank you everyone who helped, I now have an idea of a workaround ########### amd64 tests of the same disk # gpart show da4 => 40 1953525088 da4 GPT (932G) 40 532480 1 efi (260M) 532520 2008 - free - (1.0M) 534528 4194304 2 freebsd-swap (2.0G) 4728832 4194304 4 freebsd-swap (2.0G) 8923136 4194304 5 freebsd-swap (2.0G) 13117440 4194304 6 freebsd-swap (2.0G) 17311744 2048 - free - (1.0M) 17313792 8384512 7 freebsd-ufs (4.0G) 25698304 2048 - free - (1.0M) 25700352 1927823360 3 freebsd-zfs (919G) 1953523712 1416 - free - (708K) # newfs -U /dev/da4p7 /dev/da4p7: 4094.0MB (8384512 sectors) block size 32768, fragment size 4096 using 7 cylinder groups of 625.22MB, 20007 blks, 80128 inodes. with soft updates super-block backups (for fsck_ffs -b #) at: 192, 1280640, 2561088, 3841536, 5121984, 6402432, 7682880 # mount /dev/da4p7 /mnt # cd /mnt [a] #### all the following tests were run at roughly the same time, in different terminal windows. 2x git (ufs) & 2x dd. # time -l git clone https://git.freebsd.org/src.git Cloning into 'src'... remote: Enumerating objects: 4437205, done. remote: Counting objects: 100% (383246/383246), done. remote: Compressing objects: 100% (28769/28769), done. remote: Total 4437205 (delta 377514), reused 354477 (delta 354477), pack-reused 4053959 Receiving objects: 100% (4437205/4437205), 1.59 GiB | 6.10 MiB/s, done. Resolving deltas: 100% (3526142/3526142), done. Updating files: 58% (57132/97280) /mnt: write failed, filesystem is full error: unable to write file sys/contrib/dev/acpica/components/utilities/utxface.c Updating files: 58% (57317/97280) /mnt: write failed, filesystem is full Updating files: 100% (97280/97280), done. fatal: unable to checkout working tree warning: Clone succeeded, but checkout failed. You can inspect what was checked out with 'git status' and retry with 'git restore --source=HEAD :/' 639.76 real 566.78 user 41.47 sys 1054240 maximum resident set size 2566 average shared memory size 222 average unshared data size 127 average unshared stack size 475518 page reclaims 0 page faults 0 swaps 3 block input operations 15556 block output operations 422759 messages sent 1527197 messages received 606 signals received 3656856 voluntary context switches 851907 involuntary context switches # # time -l git clone https://git.freebsd.org/ports.git Cloning into 'ports'... remote: Enumerating objects: 6046014, done. remote: Counting objects: 100% (936/936), done. remote: Compressing objects: 100% (120/120), done. remote: Total 6046014 (delta 923), reused 816 (delta 816), pack-reused 6045078 Receiving objects: 100% (6046014/6046014), 1.16 GiB | 5.28 MiB/s, done. Resolving deltas: 100% (3650283/3650283), done. /mnt: write failed, filesystem is full fatal: sha1 file '/mnt/ports/.git/objects/pack/tmp_rev_cQdim5' write error. Out of diskspace fatal: fetch-pack: invalid index-pack output 553.87 real 0.41 user 1.47 sys 6092 maximum resident set size 2400 average shared memory size 208 average unshared data size 118 average unshared stack size 451 page reclaims 0 page faults 0 swaps 4 block input operations 2 block output operations 152484 messages sent 414280 messages received 0 signals received 409098 voluntary context switches 571 involuntary context switches # # time -l dd if=/dev/urandom of=/dev/da4p4 bs=8k conv=sync status=progress dd: /dev/da4p4: end of device2040 MiB) transferred 660.001s, 3242 kB/s 262145+0 records in 262144+0 records out 2147483648 bytes transferred in 660.734537 secs (3250146 bytes/sec) 660.87 real 0.44 user 10.61 sys 1992 maximum resident set size 20 average shared memory size 8 average unshared data size 132 average unshared stack size 133 page reclaims 0 page faults 0 swaps 0 block input operations 262145 block output operations 0 messages sent 0 messages received 660 signals received 262424 voluntary context switches 2182 involuntary context switches # # time -l dd if=/dev/urandom of=/dev/da4p5 bs=8k conv=sync status=progress dd: /dev/da4p5: end of device2047 MiB) transferred 666.010s, 3223 kB/s 262145+0 records in 262144+0 records out 2147483648 bytes transferred in 666.162391 secs (3223664 bytes/sec) 666.16 real 0.57 user 10.25 sys 1992 maximum resident set size 21 average shared memory size 8 average unshared data size 138 average unshared stack size 133 page reclaims 0 page faults 0 swaps 0 block input operations 262145 block output operations 0 messages sent 0 messages received 666 signals received 262420 voluntary context switches 2867 involuntary context switches # # gstat -spod dT: 1.007s w: 1.000s L(q) ops/s r/s kB kBps ms/r w/s kB kBps ms/w d/s kB kBps ms/d o/s ms/o %busy Name 1 1096 0 0 0 0.0 1096 19 20876 2.2 0 0 0 0.0 0 0.0 107.2| da4 [...] 2 1252 0 0 0 0.0 1252 8 10020 1.5 0 0 0 0.0 0 0.0 100.0| da4 When the simultaneous write test completes, swap speed in an amd64 context reverts to normal, without any user intervention: # time -l dd if=/dev/urandom of=/dev/da4p5 bs=8k conv=sync status=progress dd: /dev/da4p5: end of device2046 MiB) transferred 200.020s, 11 MB/s 262145+0 records in 262144+0 records out 2147483648 bytes transferred in 200.166970 secs (10728462 bytes/sec) 200.16 real 0.45 user 9.69 sys 2016 maximum resident set size 16 average shared memory size 6 average unshared data size 106 average unshared stack size 130 page reclaims 0 page faults 0 swaps 0 block input operations 262145 block output operations 0 messages sent 0 messages received 200 signals received 262147 voluntary context switches 320 involuntary context switches