From nobody Fri Dec 22 04:19:45 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SxDbl1Dgdz54hwx for ; Fri, 22 Dec 2023 04:20:03 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic314-21.consmr.mail.gq1.yahoo.com (sonic314-21.consmr.mail.gq1.yahoo.com [98.137.69.84]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4SxDbj5zdHz3TdS for ; Fri, 22 Dec 2023 04:20:01 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b="m/gXu85Y"; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.69.84 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1703218799; bh=3i2gS91VcrMRRwnSEn+CMKOeq7k6AhPZXFByDfjCWnw=; h=From:Subject:Date:Cc:To:References:From:Subject:Reply-To; b=m/gXu85Ygi0768nOwRXrJZjU+e8k8IKsQFKXuwRfeLpMqAkMIyZb+DsC2+UtSoQTVU4V8YB4b7g/7eaYkEk5RGulUVpCD4T8ci6la15cO3eJEJVJMbbSi0JyOyp1y0r0dx3wQ24ckW1gUCk5zQqv3Q61n/00RR4QHAYSlMIa3Im4K4soeMfPGDOHdSMyG96nflEr2bvsEKuTfan9qu/oCFh6s8XWxmDNR+SX30kkT07QbTAnVh1prED7PWrcd9kMwpJNhscaLwzqVMp5ZosRRlTybzO4XuRquAJYh3Su1p+7Sp8AcXQIxWlJ7024Z15oboyki6RWq/jy4e9tbC9Qag== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1703218799; bh=DmUl4qoi6ZpvXbHf+ah1s3bWQOCUE86wfZWTfZSBj6G=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=QUp/7yE38e25tv4HFSgaWrxxtKnIrAND85KQJB3M4zuviSef/64q7G2HfqDPzgvhDcSeM4GMs9ShjojNZ2SDmBr1oDoo+jC72zwNBn9ACWaAbpHn7ow1hRrCReU5E22AXebwCZl5P1/V8wzC3oktkBWSWLFbDqI+NBPOWYMbutAFogpGxGTxGUSHDBFXewTfwx/fs6NS17ZPk6RaGcVNYYJNMALfYAPWcqHKcJhDx7dgDd2ccpUiaGE42kzGOLwm0aTdhT27/3kHd+kyz5Csb/GJnPx1XrDSDRspWiysxckXM2ids5ldIU+pFMsoU3IlUI9FZQ7XST2R9yufAhHE5Q== X-YMail-OSG: GH1WkWcVM1lRD99ra.._XgRD_L7.f51QoHAK0wMfugm6j1nz62BTHta36PZ4iCI ZjTRFf.p4sEavLrZU.sWk.nFa_A6HZe628tB2SGX1svQY4Qa5mifFHoFv_KWie4AHLqz6yEJPBfj Wq1fkebFub_0BdjFYxFCpmSK7M1cmKt46NECt2MxnwAWpsseWDai1j48Cp8yJXsFvsidqPYFZx.H NI_tH5laslL3o8zruYywjtOwP9mI.4ixhtkuqrwNkKMcfbTAhx3S_xFA07n3il.4o4ui0FRs6qI4 DLPu_RiYvbwA6aay.TjMm1UskFgDWGmLBRkxPtIvC3OCFqZ6TzJGPezYct0ycNNSwdB3vEqr3asA TcC6gIyF2816Igrv1LnO8teBh9jU4vM0p0YVuwebVWy1lhDOX8E19WD6OsgrmOqWMIk9V5.AHfy0 RyiCfdMjgejlXSUqg7jtc2BCJCQwk86nzBjwdZvKaQGX7k6F75j9FuR9Z5nd7_ULlr8adziRX3n5 C8TtgEbRuDVDubzTGfk8BsZa6cZsDSgE0kNBcf3h5.cd8VLVI1YUyBA5Xdk3Mvijn27EFD0N0qcZ Fjd_1oi2leTRJZFynld.h2T0JhsQjKSMjAwThNJwyDZF3YGnzuWaUroleYIhysCTBUI.kf44FxMZ 2eTB1IvffIBWNym8yp20HkfAnuDV9OFjIKY2M43iBOB3Q4T8OkI0p24R0Pj5UodmXGj2wyvK39Xn gQfeZOzKuT94VD9nBID40dNM6k6gpLTtnW2UW9kGvUwse5m5BjDnN5QyMo9g1Ewjg42HFAL5_I4q LZpOOPQV2iGrR5aiqZku77p3QsShHxkNv796zwEt1E6Vs4fbmFlK_3eUsMQoyxvzxcoWNNy7Fvoy FDZLGW9dvxqPJKJb_0qqZWr3TIfPi8bSfH43nXC9ZRp9Ws.2Ic76LRz6vE13PlXboXXbh_WAFVve IhVOSKcXsE2PshirsuTWVItu4ucdcOHOHRXqTNH4pjV6fYR6UPwIwbaC9P6ejA04uESXNJ39DPLu Z1IJW1WKScTQP_eCwuQ_r2r0MqMqKoXdH4a40dEAX3QJ9lmI2gSY9zl7Cb9R8RmHiZl0LS2ytCJa iHP72TyXzMPHyBwCRFufAmUpgi8lisuTT5vP74KzlHeB2UVCsjXb_te7m65fABlQZetTEa7uqpM0 X3hIJK8cWZpl6JQ3dOjXnM4dHgUFTsDYEgN_h3zcj4xs4vLpzCvi56rWAqGjQQ.9NbhFMzmxeowI Sgd.JV6KksZfAtwT5f7DTHBiYu.NSWjQXy6o9TmXF6yrIzpSlR9XEFK0dHNI0IaEyTpnENL0f9VU RQreckUuDET.wp8v4DT87kkg2oeFnQt.84hQzx5UjRErTaPpeFmxznOyArkS6Id6DvkeU9weyvtX f5qbn0holorJ1YVckVwd.KNy6Vzqp55hNEQMFUs4__jTspfxYMx3zNi3n_NZ2iSudfcqedFxSqAu K7Laqexm02udcQjH.X9YubP_HOfqMv8fV6wOsISglBjtZrr0DukZPmFzP2VD0ao6dxbzUu3Kx9jL rfkmSTEUrHPanaSjYYxEGu.IRTsAWTlhoe10ikg_FfAQV.EncgLZ2nwMbMmM8pUPiFOujoZ3fAKX 35QmDj948_VuDg20iI6sYbrxlWs.OS46MmgEWG043WVoJssaV58Wc2C52jRsHySmUch_oAjz9RRS RtQIwv0F799wwQu0zCNg5e_8csyK2smK23PgXJoW2_H46HGxuQD6c_AuiFMqtnl0ztH094SwN7lX D3Ea2cV5Amvn99QikfNpA4s2nGFQ8qX3ftyR1g9_RyRDE6EJPlisO_6dNY72Pp23eVCFy5oQn8pe HUVVB7nzukXfSqSi0.PKWjn2C6KH5cAz97ZuRd0qeDbQvofk0JzbbbaycQiihBWvwgVI1Kfi3QAl 3UsD1tIFebgKImUE.MIYiCyooxRvJR8.FQs1ItkG6R9OxkRR56OLnOrBzpoeRwx6C2BWxVa4bkBN wbew1HMYnpLrDxaZjbDhxJHR4Q1uFt8l3vewbds8OTZhM8cuXJwN93SAPuVvR87s8FqgtWc9c7iL hizc4esbJeCewfPm9n_UpVs0XZhDxfltO3MzaHMAnK9cje3NVI3.XxiQaqDU95wEmNhwiNUfnNjh Xy9LP6i3WOEOeuhjsk2Myod7o9bp2IjQjWjHcZzzYDBIqiL3MzzEj.ze.GOqnWGvHIhCMejZCZS2 qgA1wRCZkQ4wA1IYvkMp6OW1V2GTQYQYQGHp3QF0g.suWWUHXeVZ75iw4ediI7r0MH1TivffXg5B AJmSADUbJ X-Sonic-MF: X-Sonic-ID: 73e56ac1-fbac-4d5b-aee6-a3f23b6756c4 Received: from sonic.gate.mail.ne1.yahoo.com by sonic314.consmr.mail.gq1.yahoo.com with HTTP; Fri, 22 Dec 2023 04:19:59 +0000 Received: by hermes--production-gq1-6949d6d8f9-qkzts (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 8f12744bf7d5972e2fabfb8040e97407; Fri, 22 Dec 2023 04:19:56 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.300.61.1.2\)) Subject: Re: measuring swap partition speed Message-Id: Date: Thu, 21 Dec 2023 20:19:45 -0800 Cc: Warner Losh To: void@f-m.fm, freebsd-fs@freebsd.org X-Mailer: Apple Mail (2.3774.300.61.1.2) References: X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; MV_CASE(0.50)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; ARC_NA(0.00)[]; MLMMJ_DEST(0.00)[freebsd-fs@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[98.137.69.84:from]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.69.84:from]; DKIM_TRACE(0.00)[yahoo.com:+]; FREEMAIL_FROM(0.00)[yahoo.com]; TO_DN_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FREEMAIL_TO(0.00)[f-m.fm,freebsd.org]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4SxDbj5zdHz3TdS X-Spamd-Bar: --- void wrote on Date: Fri, 22 Dec 2023 00:19:39 UTC : > Hi Mark & Warner, >=20 > I tried using fio with Warner's suggestions as a template, and > pasted the results in the latter part of this message. >=20 > On Thu, 21 Dec 2023, at 21:03, Mark Millard wrote: >=20 > . . . >=20 > fio output follows. No idea yet if the settings used are suitable for = this context > and would welcome suggestions. I later supply the output from running the same command, be it suitable or not. Again: the context is my old USB3 SSD stick, not spinning rust. Also my particular usual arm clocking configuration for the RPi4B. (I forgot to mention earlier that it has a case with a fan and heatsinks.) Also, this was not a swap space test command. So it is relevant that the file system involved was UFS, not ZFS. # swapinfo Device 1K-blocks Used Avail Capacity /dev/label/growfs_swap 8388604 0 8388604 0% I'll note that monitoring with top showed that the --numjobs=3D8 lead to 8 threads (twice the number of hardware threads). I'll note that the run left behind: # ls -C1 randread.*.0randread.0.0 randread.1.0 randread.2.0 randread.3.0 randread.4.0 randread.5.0 randread.6.0 randread.7.0 that I later deleted. > I think --filename can be a device? Output suggests > it used hw.ncpu instead of --numjobs=3D8 >=20 > ~~~~~~~ > # fio --name=3Drandread --ioengine=3Dposixaio --rw=3Drandread = --direct=3D1 --bs=3D8k --refill_buffers --norandommap --randrepeat=3D0 = --iodepth=3D32 --numjobs=3D8 --runtime=3D60 --group_reporting --thread = --size=3D2048M > randread: (g=3D0): rw=3Drandread, bs=3D(R) 8192B-8192B, (W) = 8192B-8192B, (T) 8192B-8192B, ioengine=3Dposixaio, iodepth=3D32 > ... > fio-3.36 > Starting 8 threads > randread: Laying out IO file (1 file / 2048MiB) > randread: Laying out IO file (1 file / 2048MiB) > randread: Laying out IO file (1 file / 2048MiB) > randread: Laying out IO file (1 file / 2048MiB) > randread: Laying out IO file (1 file / 2048MiB) > randread: Laying out IO file (1 file / 2048MiB) > randread: Laying out IO file (1 file / 2048MiB) > randread: Laying out IO file (1 file / 2048MiB) > Jobs: 4 (f=3D4): = [_(1),r(2),_(2),r(1),_(1),r(1)][21.2%][r=3D288KiB/s][r=3D36 IOPS][eta = 04m:05s] > randread: (groupid=3D0, jobs=3D8): err=3D 0: pid=3D135125: Thu Dec 21 = 16:43:00 2023 > read: IOPS=3D43, BW=3D351KiB/s (359kB/s)(22.6MiB/65986msec) > slat (nsec): min=3D889, max=3D1948.6k, avg=3D4172.45, stdev=3D36488.07 > clat (msec): min=3D1108, max=3D11660, avg=3D5644.23, stdev=3D1282.12 > lat (msec): min=3D1108, max=3D11660, avg=3D5644.24, stdev=3D1282.12 > clat percentiles (msec): > | 1.00th=3D[ 1183], 5.00th=3D[ 3171], 10.00th=3D[ 4933], 20.00th=3D[ = 5269], > | 30.00th=3D[ 5470], 40.00th=3D[ 5604], 50.00th=3D[ 5738], 60.00th=3D[ = 5873], > | 70.00th=3D[ 5940], 80.00th=3D[ 6074], 90.00th=3D[ 6342], 95.00th=3D[ = 6812], > | 99.00th=3D[10671], 99.50th=3D[10939], 99.90th=3D[11610], = 99.95th=3D[11610], > | 99.99th=3D[11610] > bw ( KiB/s): min=3D 208, max=3D 3760, per=3D100.00%, avg=3D1535.05, = stdev=3D128.23, samples=3D245 > iops : min=3D 26, max=3D 470, avg=3D191.88, stdev=3D16.03, samples=3D245= > lat (msec) : 2000=3D3.25%, >=3D2000=3D96.75% > cpu : usr=3D0.00%, sys=3D0.12%, ctx=3D22712, majf=3D0, minf=3D0 > IO depths : 1=3D0.3%, 2=3D0.6%, 4=3D1.1%, 8=3D4.9%, 16=3D69.6%, = 32=3D23.6%, >=3D64=3D0.0% > submit : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%, 32=3D0.0%, = 64=3D0.0%, >=3D64=3D0.0% > complete : 0=3D0.0%, 4=3D94.7%, 8=3D2.9%, 16=3D1.5%, 32=3D0.9%, = 64=3D0.0%, >=3D64=3D0.0% > issued rwts: total=3D2895,0,0,0 short=3D0,0,0,0 dropped=3D0,0,0,0 > latency : target=3D0, window=3D0, percentile=3D100.00%, depth=3D32 >=20 > Run status group 0 (all jobs): > READ: bw=3D351KiB/s (359kB/s), 351KiB/s-351KiB/s (359kB/s-359kB/s), = io=3D22.6MiB (23.7MB), run=3D65986-65986msec # fio --name=3Drandread --ioengine=3Dposixaio --rw=3Drandread --direct=3D1= --bs=3D8k --refill_buffers --norandommap --randrepeat=3D0 --iodepth=3D32 = --numjobs=3D8 --runtime=3D60 --group_reporting --thread --size=3D2048M randread: (g=3D0): rw=3Drandread, bs=3D(R) 8192B-8192B, (W) 8192B-8192B, = (T) 8192B-8192B, ioengine=3Dposixaio, iodepth=3D32 ... fio-3.36 Starting 8 threads randread: Laying out IO file (1 file / 2048MiB) randread: Laying out IO file (1 file / 2048MiB) randread: Laying out IO file (1 file / 2048MiB) randread: Laying out IO file (1 file / 2048MiB) randread: Laying out IO file (1 file / 2048MiB) randread: Laying out IO file (1 file / 2048MiB) randread: Laying out IO file (1 file / 2048MiB) randread: Laying out IO file (1 file / 2048MiB) Jobs: 8 (f=3D8): [r(8)][100.0%][r=3D17.4MiB/s][r=3D2222 IOPS][eta = 00m:00s] randread: (groupid=3D0, jobs=3D8): err=3D 0: pid=3D100241: Sun Dec 17 = 12:42:53 2023 read: IOPS=3D2486, BW=3D19.4MiB/s (20.4MB/s)(1168MiB/60125msec) slat (nsec): min=3D888, max=3D1494.1k, avg=3D3446.83, stdev=3D8015.07 clat (msec): min=3D58, max=3D334, avg=3D102.70, stdev=3D22.68 lat (msec): min=3D58, max=3D334, avg=3D102.70, stdev=3D22.68 clat percentiles (msec): | 1.00th=3D[ 70], 5.00th=3D[ 75], 10.00th=3D[ 79], = 20.00th=3D[ 84], | 30.00th=3D[ 89], 40.00th=3D[ 94], 50.00th=3D[ 99], = 60.00th=3D[ 105], | 70.00th=3D[ 111], 80.00th=3D[ 118], 90.00th=3D[ 136], = 95.00th=3D[ 150], | 99.00th=3D[ 171], 99.50th=3D[ 180], 99.90th=3D[ 211], = 99.95th=3D[ 228], | 99.99th=3D[ 268] bw ( KiB/s): min=3D 8246, max=3D24640, per=3D100.00%, avg=3D19903.38, = stdev=3D281.80, samples=3D960 iops : min=3D 1028, max=3D 3080, avg=3D2487.90, stdev=3D35.24, = samples=3D960 lat (msec) : 100=3D52.92%, 250=3D47.07%, 500=3D0.01% cpu : usr=3D0.17%, sys=3D5.01%, ctx=3D1016480, majf=3D0, = minf=3D0 IO depths : 1=3D0.1%, 2=3D0.1%, 4=3D0.1%, 8=3D0.7%, 16=3D75.3%, = 32=3D24.0%, >=3D64=3D0.0% submit : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%, 32=3D0.0%, = 64=3D0.0%, >=3D64=3D0.0% complete : 0=3D0.0%, 4=3D93.2%, 8=3D4.7%, 16=3D2.0%, 32=3D0.1%, = 64=3D0.0%, >=3D64=3D0.0% issued rwts: total=3D149497,0,0,0 short=3D0,0,0,0 dropped=3D0,0,0,0 latency : target=3D0, window=3D0, percentile=3D100.00%, depth=3D32 Run status group 0 (all jobs): READ: bw=3D19.4MiB/s (20.4MB/s), 19.4MiB/s-19.4MiB/s = (20.4MB/s-20.4MB/s), io=3D1168MiB (1225MB), run=3D60125-60125msec After swapoff instead: # fio --name=3Drandread --ioengine=3Dposixaio --rw=3Drandread --direct=3D1= --bs=3D8k --refill_buffers --norandommap --randrepeat=3D0 --iodepth=3D32 = --numjobs=3D8 --runtime=3D60 --group_reporting --thread --size=3D2048M randread: (g=3D0): rw=3Drandread, bs=3D(R) 8192B-8192B, (W) 8192B-8192B, = (T) 8192B-8192B, ioengine=3Dposixaio, iodepth=3D32 ... fio-3.36 Starting 8 threads randread: Laying out IO file (1 file / 2048MiB) randread: Laying out IO file (1 file / 2048MiB) randread: Laying out IO file (1 file / 2048MiB) randread: Laying out IO file (1 file / 2048MiB) randread: Laying out IO file (1 file / 2048MiB) randread: Laying out IO file (1 file / 2048MiB) randread: Laying out IO file (1 file / 2048MiB) randread: Laying out IO file (1 file / 2048MiB) Jobs: 8 (f=3D8): [r(8)][100.0%][r=3D17.6MiB/s][r=3D2259 IOPS][eta = 00m:00s] randread: (groupid=3D0, jobs=3D8): err=3D 0: pid=3D100211: Sun Dec 17 = 12:07:16 2023 read: IOPS=3D2471, BW=3D19.3MiB/s (20.2MB/s)(1161MiB/60119msec) slat (nsec): min=3D889, max=3D1912.6k, avg=3D3415.21, stdev=3D8521.17 clat (msec): min=3D54, max=3D762, avg=3D103.32, stdev=3D28.95 lat (msec): min=3D54, max=3D762, avg=3D103.32, stdev=3D28.95 clat percentiles (msec): | 1.00th=3D[ 69], 5.00th=3D[ 75], 10.00th=3D[ 79], = 20.00th=3D[ 84], | 30.00th=3D[ 89], 40.00th=3D[ 93], 50.00th=3D[ 99], = 60.00th=3D[ 105], | 70.00th=3D[ 110], 80.00th=3D[ 117], 90.00th=3D[ 136], = 95.00th=3D[ 150], | 99.00th=3D[ 174], 99.50th=3D[ 188], 99.90th=3D[ 518], = 99.95th=3D[ 558], | 99.99th=3D[ 617] bw ( KiB/s): min=3D12336, max=3D24912, per=3D100.00%, avg=3D19913.12, = stdev=3D267.11, samples=3D954 iops : min=3D 1542, max=3D 3114, avg=3D2489.14, stdev=3D33.39, = samples=3D954 lat (msec) : 100=3D52.77%, 250=3D47.01%, 500=3D0.11%, 750=3D0.11%, = 1000=3D0.01% cpu : usr=3D0.18%, sys=3D4.93%, ctx=3D1005861, majf=3D0, = minf=3D0 IO depths : 1=3D0.1%, 2=3D0.1%, 4=3D0.3%, 8=3D1.3%, 16=3D74.8%, = 32=3D23.6%, >=3D64=3D0.0% submit : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%, 32=3D0.0%, = 64=3D0.0%, >=3D64=3D0.0% complete : 0=3D0.0%, 4=3D93.4%, 8=3D4.5%, 16=3D1.9%, 32=3D0.2%, = 64=3D0.0%, >=3D64=3D0.0% issued rwts: total=3D148600,0,0,0 short=3D0,0,0,0 dropped=3D0,0,0,0 latency : target=3D0, window=3D0, percentile=3D100.00%, depth=3D32 Run status group 0 (all jobs): READ: bw=3D19.3MiB/s (20.2MB/s), 19.3MiB/s-19.3MiB/s = (20.2MB/s-20.2MB/s), io=3D1161MiB (1217MB), run=3D60119-60119msec The difference: Jobs: 4 (f=3D4) (yours) vs. Jobs: 8 (f=3D8) (mine) is not obvious, given the same commands. =3D=3D=3D Mark Millard marklmi at yahoo.com