From nobody Sun May 21 10:48:13 2023 X-Original-To: freebsd-arm@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QPHPH1D5fz4CLnW for ; Sun, 21 May 2023 10:48:35 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic311-23.consmr.mail.gq1.yahoo.com (sonic311-23.consmr.mail.gq1.yahoo.com [98.137.65.204]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4QPHPF6Wycz46tR for ; Sun, 21 May 2023 10:48:33 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=GvYYDnoE; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.65.204 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1684666112; bh=LwucKUFf3yx7FTxLcnUodCBZtUs4hCInldtx+o6X+C8=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=GvYYDnoENv2PtgvlR6WkLAA4Lq4hOpOxAVvKc/cwpoaow5y/XjGZessWWCuiGKLrGZBKYEObMT1wvCtnDB7cyW0/p5VtwIWf9tpE2frW0q2ikEyp/DKd+aHwNnRZQPxAgPUyRynw1lr/yrURLbqwOu+CD/XyeJNnX/4MOVU+f72g1CFvWB/XHF5ydNysN1nErhCpCZ8m58HghvhFnfRFdsyMaL80byAjxXBuZ+5jCr8ymiPWULazVNTlYhaI8VuZETHtc0ljHpjqEGqb0s+KRTgARxGt6L8HJJ4RrUH6JvXH4BbQwKhvG5/1B7BEf+wDo9kjMDwQrCOYMAI6Q57BRA== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1684666112; bh=1HULwSYVapwA0OVqBYVr5YfnH7/WbhXw1Wfj0J7LZZ5=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=MhNFt3auJAWK91BwzOPPNXtUWR1ddNCiGb0bUFUR/jrS6lPAz6TKjF15Fx80Iwdt5/e9YB8o4GeDE1LH/Gh4a+0RLXMEzarfHUAOI8W+nmO4bB44T9Oto/L+hGwva9o8BhBtoHlei8DxaiKjtgXxKKrwVwU4YyMe3P4kJi7I4zP9gkUuxbfFZvhIw6QROQR/n2izjiirFk6m+nWgGDanT6u/X7S2m60rN5qMnggnt/bGVxUO7TC78mgBOZmx3/ILWj7fFi03w5SaOHxkw/so4NWYFawAEP76B/4YjdsP65RKdQK/RFMdQVNtWNnTjT7zNFEJY9hSjgFE/f492qE3/w== X-YMail-OSG: VLi9NbcVM1k3Okx2YtbBW55Zv4OzMyJ8BuFR7Q7Wm93Lt5INrpAorBLGjUzixie XQNzWKx_iPkUQ3W7Csk6KWzmRgtGlznjuD0OQ0H9F_frfnaa_YFS5XuCNM79_B9eKsW9USFtgY1y cVUiEDD7sbw7DXORpZOtkZz2LICeDtJ_MD1OG8Mp_rwW1nEKDg8QEAyEbeyi56XoeF36cFGCBDfE CEgIYSD.aqv_FCao1treZix5fGZ7P9EuUepxds9EgOenWMjUwnxzydSQ14ngqERuQ8KMAhAYo1P9 wqQ7mJKBIJITyynWV8BGrAnT0W3qzw08mV80sODTRtkOkZ30oyo8dqkwUUhmmKh3zeodWM2HTR6p 7VPSc1QUHel__UBM0L1Ox6Y_rUO8V3_vabWGJvjgA1gtkGkeXy71IfaZnMZRN5UK7jOtzjA7j9p4 TKmoh4f7KxVstVVVcXVUyqdL6OdjB3sCp944D6uOofETb.jHd_l16JR.3AxR0mYBP9qS3SkR2ola P2LoHH3YFFGqCshRaUo51k2aXy3RhaPYZ48hWaf48VUj498u3mO5Z13eIbcKfTxxoY46aouLyqqt 46_YeTDcZ7ZmJWAtaMsQrVXPZjXcNyvvtnXMEIdHbeQmSJo6kDswviLn_oXiWB47HIiDJNC.XwVO dBsrJ4oVenj3H_QJD_VSVbToKqMRL1Nv5GWGmrQ84fdw8fQOZGKZzgED7MfCMbrRZ25nNXA8rsT6 4j42p0pnq6fGx8hBxtNC2uUpvwSJYK6rRlvgd9otHn7NHf8ZYqVcB6QU_X28uHrWf555S32L31ko DoAfCnoVlD7ig17o0pqGUzwuJH_ry46iTs8CHCNmhv_PCW9GuOwfCsrsLpoUpN7AM9WdDkLyi4L6 0OJv9zsXp5CXS7OFD09aj766aILZm7OQ.mSSWuqTAb4N97aplnc_zobQyTDGB4Mzpu2w1Zin7vb3 fl_sGGx0zwOCRsM8AsN1LsZcoIKkNhHzvGsk1Letf.74DTJGita5k3kPfhiI_M9Ele34SUTHHboM IlPw0tBTFmt8UTua.COGMX6xBeUZNabWFEsX4OuP1LxjEUvx_C_Xu2Vt9TJLl7wAooDl197J2HtY ZE4UVvzkUq4KkFZRbUzc_jIc3Gsia.6uXVG65gAeS8.qbRN1JRW0V_51JtRpbpHSEiJ.I2QT5h5F 7l6dGHEcINAx30Cd7L5vvmodwS_z44HhkoXikFznLayWcxhIEBPC6sACGs9sjv.Sb0CCMPpon8Q0 GnzrVrZ3iBhz6Mtb4Isq1M9_teaiCAX.SK9_hstqxsHH7jf6lwxzU3toA8MufkI9.A3OO61h017v Rxkmz2q6rs1TYU28W53vIUFfBMLsdo1b7TfLUuxZZDiY4B72SxsxZkToKiyX9Zm0Lpc6Ak38MgWC SOEEmsA1y2PhtoZGbYCLIDkyfRrV_hjr_S1DgcjdyBopJT4CgTY6J9wnt9QuiwHRXEo1W48s7Mt2 CrbPgpEHoYwotHr0HcbMbBTeORvlbeK_c4WSgXBoFDLqVCq8Pe.QTGAth2g2FEvzuobu7fnEji53 bGUMZjhDzNG31UIBB9orvKZ3G0utN76pDDmkUkM42HsICwKfofvyuhHelbIbpHDar7T.1f3KE_5s bPHviLDs.bgathwSPpB05uU.UiIZtLAP8Ucsnv7iO1GZNNA0edIMN12eJWnN9rsl6vyR8_Ue.fmJ OW0zPwOG2uMieRlBoMXnguz6rKGt8XFj2JFPpCuuE6QvSEpLsTA.4nbnTsof.96yoavr2DSIz_ms FURGobQrgkQ5EVBjq2psEzC4zWyBfLx1KIwqnkcShuhp8p8z4kGkgSJHsxVhidwc8xttnvYJ1o2K SawcoKHWyESCIKh6xbEtZ9uJPmGaV0YT6Fz9S5V.EQdGK_u_RCXCLxRGdNnvnI4SAiIKer1fw2f8 2g5klHmWf3lVGCObKq2_OUp_aXGvg9kA1jwDY.yIHsvTUSlXsOGuiyHdt4RJ5oPAUwcwLBplglrK CaQUXkxo2BHgqDumJafdtEUXfB33DbT3bxWL1Hh0K50OrfMHNpa3JC.55W0y90LYY8KwxNEK5V1U J8ipdIqTOlFSTMDFm002ZCSq3KcWCF2KQoHZdi9ka2fcfk7RI_sA56hb__4yl4GFtNuXSYSwQ98f oalnVpwZSuLCt7nMOL0zR7zL5AuAzlJ7frTslmwwS_CLc7Ohk3QAMe_Is3XG7391v8EFvbUKp0Ju os_Qsy_j8Rv7dHRphflAaCl.VaLt0VjuSZpZxQ0Bw6fZpbrldqrjzCH1C_r7eWFpzN73NPssUxXH K X-Sonic-MF: X-Sonic-ID: 6675c8ea-2e8a-4555-b6ad-237ed87e5dc9 Received: from sonic.gate.mail.ne1.yahoo.com by sonic311.consmr.mail.gq1.yahoo.com with HTTP; Sun, 21 May 2023 10:48:32 +0000 Received: by hermes--production-bf1-54475bbfff-xvhwr (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 12ca2e9323996b10839030965a194653; Sun, 21 May 2023 10:48:25 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Porting FreeBSD to ARM processors List-Archive: https://lists.freebsd.org/archives/freebsd-arm List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arm@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Re: Armv7 (rpi2) getting stuck in buildworld for -current From: Mark Millard In-Reply-To: <7AA87E52-AB3D-48E4-B62A-EE73EBB23003@yahoo.com> Date: Sun, 21 May 2023 03:48:13 -0700 Cc: freebsd-arm Content-Transfer-Encoding: quoted-printable Message-Id: <3C1F1156-F0CA-4958-B2D1-A28B2F2D36F5@yahoo.com> References: <7758885B-3115-47F0-A453-1C010313D4B8@yahoo.com> <03F330A1-35E4-40D4-B9C6-407041BBEC58@yahoo.com> <7AA87E52-AB3D-48E4-B62A-EE73EBB23003@yahoo.com> To: bob prohaska X-Mailer: Apple Mail (2.3731.400.51.1.1) X-Spamd-Result: default: False [-1.58 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_SPAM_SHORT(0.92)[0.922]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; RCVD_IN_DNSWL_NONE(0.00)[98.137.65.204:from]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; TO_DN_ALL(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[yahoo.com]; MLMMJ_DEST(0.00)[freebsd-arm@freebsd.org]; DKIM_TRACE(0.00)[yahoo.com:+]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; MIME_TRACE(0.00)[0:+]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.65.204:from] X-Rspamd-Queue-Id: 4QPHPF6Wycz46tR X-Spamd-Bar: - X-ThisMailContainsUnwantedMimeParts: N On May 21, 2023, at 00:55, Mark Millard wrote: > On May 20, 2023, at 11:59, Mark Millard wrote: >=20 >> I set up the RPi2B v1.1 and started a -j4 buildworld buildkernel >> from-scratch rebuild on/of: >>=20 >> # uname -apKU # long output line split for readability >> FreeBSD OPiP2E_RPi2v1p1 14.0-CURRENT FreeBSD 14.0-CURRENT #74 >> main-n262658-b347c2284603-dirty: Fri Apr 28 23:07:41 PDT 2023 >> = root@CA72_16Gp_ZFS:/usr/obj/BUILDs/main-CA7-nodbg-clang/usr/main-src/arm.a= rmv7/sys/GENERIC-NODBG-CA7 >> arm armv7 1400088 1400088 >>=20 >> (The original build was done on another machine.) >>=20 >>=20 >> At somewhat under 18 hrs it finished with the large swap >> use during the overlapping time frame for these 4 builds: >>=20 >> clang/libclang/CodeGen/CodeGenAction.o >> clang/libclang/CodeGen/CodeGenFunction.o >> clang/libclang/CodeGen/CodeGenModule.o >> clang/libclang/CodeGen/CodeGenPGO.o >>=20 >> These are my notes from the information for somewhat after >> the swap use dropped off. >>=20 >>=20 >>=20 >> My armv7 builds disable targeting other architectures but >> I also have WITH_CLANG_EXTRAS=3D . (Not that the build has >> gotten that far yet.) I show the controlling file content >> later below. >>=20 >> I used no assignments for: >>=20 >> #vm.pfault_oom_attempts=3D-1 >> #vm.pfault_oom_attempts=3D 10 >> #vm.pfault_oom_wait=3D ??? >>=20 >> but did/do have: >>=20 >> vm.pageout_oom_seq=3D120 >>=20 >> vm.swap_enabled=3D0 >> vm.swap_idle_enabled=3D0 >>=20 >> in use for this experiment. >>=20 >> FYI: >> make[1]: "/usr/main-src/Makefile.inc1" line 326: SYSTEM_COMPILER: = Determined that CC=3Dcc matches the source tree. Not bootstrapping a = cross-compiler. >> make[1]: "/usr/main-src/Makefile.inc1" line 331: SYSTEM_LINKER: = Determined that LD=3Dld matches the source tree. Not bootstrapping a = cross-linker. >>=20 >> (Those 2 have significant time implications for the overall >> build.) >>=20 >> Based on (my modified) top, sampling every 3 seconds, >>=20 >> Mem: . . ., >> 754020Ki MaxObsActive, >> 186756Ki MaxObsWired, >> 923356Ki MaxObs(Act+Wir+Lndry) >>=20 >> Swap: 1740Mi Total, . . ., >> 756828Ki MaxObsUsed, >> 1442Mi MaxObs(Act+Lndry+SwapUsed), >> 1615Mi MaxObs(Act+Wir+Lndry+SwapUsed) >>=20 >> So: slightly over 739 MiBytes of swap observed to have been >> in use at one time. >>=20 >> As for the overlapping time's duration: file creation and >> modification times, in time order were: >>=20 >> (via extraction from ls -TldU output:) >> 09:37:28 creation of clang/libclang/CodeGen/CodeGenAction.o >> 09:38:44 creation of clang/libclang/CodeGen/CodeGenFunction.o >> 09:40:19 creation of clang/libclang/CodeGen/CodeGenModule.o >> 09:41:28 creation of clang/libclang/CodeGen/CodeGenPGO.o >>=20 >> (via extraction from ls -Tld output:) >> 09:47:15 modification of clang/libclang/CodeGen/CodeGenFunction.o >> 09:49:53 modification of clang/libclang/CodeGen/CodeGenAction.o >> 09:50:10 modification of clang/libclang/CodeGen/CodeGenPGO.o >> 09:54:49 modification of clang/libclang/CodeGen/CodeModule.o >>=20 >> So: >>=20 >> 09:41:28 . . . 09:47:15 (under 6 min) for the overlapping time >> frame and the highest swap space use happened inside that >> interval. >>=20 >> During this, there were times mixes of CPUn and "swread" STATE >> for the compiles. But at no point were all observed to be >> blocked waiting, at no point was only 1 observed to show a CPUn >> with a large WCPU. >>=20 >> This is largely attributable to the USB media having tiny >> latencies compared to spinning rust and having reasonable >> transfer rates for the type of I/O: NMVe USB3 media (that is >> also USB2 compatible for USB powered usage). >>=20 >> My use of: >>=20 >> # >> # Delay when persistent low free RAM leads to >> # Out Of Memory killing of processes: >> vm.pageout_oom_seq=3D120 >>=20 >> and: >>=20 >> # >> # Together this pair avoids swapping out the process kernel stacks. >> # This avoids processes for interacting with the system from being >> # hung-up. >> vm.swap_enabled=3D0 >> vm.swap_idle_enabled=3D0 >>=20 >> did not lead to any problems so far. >>=20 >>=20 >> For reference: >>=20 >> Via systat -vmstat I monitored . . . >>=20 >> VN PAGER SWAP PAGER >> in out in out >> count =20 >> pages >> ioflt . . . >> . . . >> intrn . . . >>=20 >> Both VN in and SWAP in can contribute to ioflt, faults >> that required I/O. The ioflt number would be before the >> "ioflt" text. >>=20 >> There is a later line that lists intrn (somewhat below >> ioflt): "in-transit blocking page faults". The intrn >> number would be before the "intrn" text. >>=20 >> The figures varied under 600 for ioflt and intrn for >> what I saw during the large swap space use, with >> matching SWAP activity, no significant VN activity. >> (The figures are for an about 5 second update interval, >> as I remember.) (I watched the on screen updates. >> I did not try to capture the material in a file.) >>=20 >> I expect that these figures would be large for a >> sustained period in your context. >>=20 >>=20 >> I also monitored with "gstat -spod". I assume that stat is >> more familiar. (I use -spod even when "d" happens to not be >> going to show any activity.) >>=20 >>=20 >> [I do not recommend leaving "systat -swap" running: it >> accumulates a large set of memory leaks and so can >> mess up tracking swap space use by being a signficant >> contributor. I did not put it to significant use other >> than discovering that problem.] >>=20 >>=20 >> Configuration points . . . >>=20 >>=20 >> /boot/efi/config.txt has: >>=20 >> enable_uart=3D1 >> dtoverlay=3Dmmc >> # >> # Local addition that avoids (at least) USB3 SSD boot failures that = look like: >> # uhub_reattach_port: port ? reset failed, error=3DUSB_ERR_TIMEOUT >> # uhub_reattach_port: device problem (USB_ERR_TIMEOUT), disabling = port ? >> initial_turbo=3D60 >> # >> # Local additions: >> uart_2ndstage=3D1 >> dtdebug=3D1 >> kernel=3Du-boot.bin.2023.01.armv7 >> kernel7=3Du-boot.bin.2023.01.armv7 >> dtoverlay=3Ddisable-bt >> # >> force_turbo=3D1 >>=20 >> ( /etc/rc.conf has powerd commented out. ) >> (I build u-boot with a couple of settings added.) >> (Leaving initial_turbo in place allows disabling >> force_turbo independently --but still allowing >> the USB booting to work during the temporary >> turbo status. intial_turbo is not required when >> force_turbo is enabled --but does not hurt.) >>=20 >> /boot/loader.conf has : >>=20 >> # >> # Delay when persistent low free RAM leads to >> # Out Of Memory killing of processes: >> vm.pageout_oom_seq=3D120 >> # >> # For plunty of swap/paging space (will not >> # run out), avoid pageout delays leading to >> # Out Of Memory killing of processes: >> #vm.pfault_oom_attempts=3D-1 >> # >> # For possibly insufficient swap/paging space >> # (might run out), increase the pageout delay >> # that leads to Out Of Memory killing of >> # processes: >> #vm.pfault_oom_attempts=3D 10 >> #vm.pfault_oom_wait=3D ??? >> # (The multiplication is the total but there >> # are other potential tradoffs in the factors >> # multiplied, even for nearly the same total.) >>=20 >> (As I understand you are now using defaults for >> vm.pfault_oom_attempts and vm.pfault_oom_wait . >> So I did as well for those 2 for this experiment.) >>=20 >>=20 >> /etc/sysctl.conf has: >>=20 >> # >> # Together this pair avoids swapping out the process kernel stacks. >> # This avoids processes for interacting with the system from being >> # hung-up. >> vm.swap_enabled=3D0 >> vm.swap_idle_enabled=3D0 >>=20 >>=20 >> # more ~/src.configs/src.conf.CA7-nodbg-clang-alt.aarch64-host=20 >> TO_TYPE=3Darmv7 >> # >> KERNCONF=3DGENERIC-NODBG-CA7 >> TARGET=3Darm >> .if ${.MAKE.LEVEL} =3D=3D 0 >> TARGET_ARCH=3D${TO_TYPE} >> .export TARGET_ARCH >> .endif >> # >> WITH_SYSTEM_COMPILER=3D >> WITH_SYSTEM_LINKER=3D >> # >> WITH_ELFTOOLCHAIN_BOOTSTRAP=3D >> #Disables avoiding bootstrap: WITHOUT_LLVM_TARGET_ALL=3D >> WITHOUT_LLVM_TARGET_AARCH64=3D >> WITH_LLVM_TARGET_ARM=3D >> WITHOUT_LLVM_TARGET_MIPS=3D >> WITHOUT_LLVM_TARGET_POWERPC=3D >> WITHOUT_LLVM_TARGET_RISCV=3D >> WITHOUT_LLVM_TARGET_X86=3D >> WITH_CLANG=3D >> WITH_CLANG_IS_CC=3D >> WITH_CLANG_FULL=3D >> WITH_CLANG_EXTRAS=3D >> WITH_LLD=3D >> WITH_LLD_IS_LD=3D >> # >> WITH_LLDB=3D >> # >> WITH_BOOT=3D >> # >> WITHOUT_WERROR=3D >> MALLOC_PRODUCTION=3D >> WITH_MALLOC_PRODUCTION=3D >> WITHOUT_ASSERT_DEBUG=3D >> WITHOUT_LLVM_ASSERTIONS=3D >> # >> # Avoid stripping but do not control host -g status as well: >> DEBUG_FLAGS+=3D >> # >> WITH_REPRODUCIBLE_BUILD=3D >> WITH_DEBUG_FILES=3D >> # >> XCFLAGS+=3D -mcpu=3Dcortex-a7 >> XCXXFLAGS+=3D -mcpu=3Dcortex-a7 >> # There is no XCPPFLAGS but XCPP gets XCFLAGS content. >>=20 >> (An armv7 host does not need differing content than an >> aarch64 host, thus the use of the *.aarch64-host file.) >>=20 >> However long the overall build ends up taking, the above >> is part of why the details end up as they will end up. >>=20 >>=20 >> /etc/crontab notes: >>=20 >> I do not know if you leave the following enabled during the long >> builds or not ( from /etc/crontab ): >>=20 >> # Perform daily/weekly/monthly maintenance. >> 1 3 * * * root periodic daily >> 15 4 * * 6 root periodic weekly >> 30 5 1 * * root periodic monthly >>=20 >> that can run things like "/usr/local/sbin/pkg check -qsa" >> (daily example) that would compete for resources. I left them >> active, so daily competed with the build for a while, but it >> did not happen to overlap with the high swapspace use time >> frame. >>=20 >> I commonly disable these for builds that will span into the >> hours it indicates, at least when I'm monitoring builds for >> comparisons and such. >>=20 >=20 > FYI: the buildworld just completed a bit ago: >=20 > World built in 117913 seconds, ncpu: 4, make -j4 >=20 > So, somewhat under 33 hrs for what and how I build, > given the media I use. >=20 > The buildkernel is in process. >=20 I see that the buildkernel finished: Kernel(s) GENERIC-NODBG-CA7 built in 6475 seconds, ncpu: 4, make -j4 So, somewhat under 2 hrs. Overall world+kernel: (117913+6475) sec, i.e., a little under 34 hr 34 min. And, overall world+kernel: Mem: . . ., 754020Ki MaxObsActive, 186756Ki MaxObsWired, 932440Ki MaxObs(Act+Wir+Lndry) Swap: 1740Mi Total, . . ., 756828Ki MaxObsUsed, 1442Mi MaxObs(Act+Lndry+SwapUsed), 1615Mi MaxObs(Act+Wir+Lndry+SwapUsed) =3D=3D=3D Mark Millard marklmi at yahoo.com