FYI: what it takes for RAM+swap to build devel/llvm40 with 4 processors or cores and WITH__DEBUG= (powerpc64 example)

Sat Apr 1 10:51:14 UTC 2017

On 2017-Mar-31, at 4:51 PM, Mark Millard <markmi at dsl-only.net> wrote:

> On 2017-Mar-30, at 7:51 PM, Mark Millard <markmi at dsl-only.net> wrote:
> 
>> On 2017-Mar-30, at 1:22 PM, Mark Millard <markmi at dsl-only.net> wrote:
>> 
>>> Sounds like the ALLOW_OPTIMIZATIONS_FOR_WITH_DEBUG technique
>>> would not change the "WITNESS and INVARIANTS"-like part of the
>>> issue. In fact if WITH_DEBUG= causes the cmake debug-style
>>> llvm40 build ALLOW_OPTIMIZATIONS_FOR_WITH_DEBUG might not
>>> make any difference: separate enforcing of lack of optimization.
>>> 
>>> But just to see what results I've done "pkg delete llvm40"
>>> and am doing another build with ALLOW_OPTIMIZATIONS_FOR_WITH_DEBUG=
>>> and its supporting code in place in addition to using WITH_DEBUG=
>>> as the type of build fro FreeBSD's viewpoint.
>>> 
>>> If you know that the test is a waste of machine cycles, you can
>>> let me know if you want.
>> 
>> The experiment showed that ALLOW_OPTIMIZATIONS_FOR_WITH_DEBUG
>> use made no difference for devel/llvm40 so devel/llvm40 itself
>> has to change such as what Dimitry Andric reported separately
>> as a working change to the Makefile .
>> 
>> (ALLOW_OPTIMIZATIONS_FOR_WITH_DEBUG would still have its uses
>> for various other ports.)
> 
> I've now tried with both ALLOW_OPTIMIZATIONS_FOR_WITH_DEBUG and:

I may have had a textual error that prevented
ALLOW_OPTIMIZATIONS_FOR_WITH_DEBUG from even potentially
contributing. So I'll re-run this test.

For now I presume that what I reported was okay and so
I continue to refer to these figures later below.

> # svnlite diff /usr/ports/devel/llvm40/
> Index: /usr/ports/devel/llvm40/Makefile
> ===================================================================
> --- /usr/ports/devel/llvm40/Makefile	(revision 436747)
> +++ /usr/ports/devel/llvm40/Makefile	(working copy)
> @@ -236,6 +236,11 @@
> 
> .include <bsd.port.pre.mk>
> 
> +.if defined(WITH_DEBUG)
> +CMAKE_BUILD_TYPE=	RelWithDebInfo
> +STRIP=
> +.endif
> +
> _CRTLIBDIR=	${LLVM_PREFIX:S|${PREFIX}/||}/lib/clang/${LLVM_RELEASE}/lib/freebsd
> .if ${ARCH} == "amd64"
> _COMPILER_RT_LIBS= \
> 
> 
> 
> pkg delete after the build reports:
> 
> Installed packages to be REMOVED:
> 	llvm40-4.0.0
> 
> Number of packages to be removed: 1
> 
> The operation will free 42 GiB.
> 
> So down by 7 GiBytes from 49 GiBytes.
> 
> (I did not actually delete it.)
> 
> Also:
> 
> # du -sg /usr/obj/portswork/usr/ports/devel/llvm40
> 102	/usr/obj/portswork/usr/ports/devel/llvm40
> 
> which is down by 16 GiBytes from 118 GiBytes.
> 
> Reminder: These are from portmaster -DK so no
> cleanup after the build, which is what leaves
> the source code and such around in case of
> needing to look at a problem.
> 
> (102+42) GiBytes == 146 GiBytes.
> vs.
> (118+49) GiBytes == 167 GiBytes.
> 
> So a difference of 21 GiBytes (or so).
> 
> But that is for everything in each case (and
> WITH_DEBUG= in use):
> 
> # more /var/db/ports/devel_llvm40/options
> # This file is auto-generated by 'make config'.
> # Options for llvm40-4.0.0.r4
> _OPTIONS_READ=llvm40-4.0.0.r4
> _FILE_COMPLETE_OPTIONS_LIST=CLANG DOCS EXTRAS LIT LLD LLDB
> OPTIONS_FILE_SET+=CLANG
> OPTIONS_FILE_SET+=DOCS
> OPTIONS_FILE_SET+=EXTRAS
> OPTIONS_FILE_SET+=LIT
> OPTIONS_FILE_SET+=LLD
> OPTIONS_FILE_SET+=LLDB
> 
> So avoiding WITH_DEBUG= and/or various build options
> is still the major way of avoiding use of lots of space
> if it is an issue.
> 
> 
> 
> Why no RAM+SWAP total report this time:
> 
> As far as I know FreeBSD does not track or report peak
> swap-space usage since the last boot. And, unfortunately
> I was not around to just sit and watch a top display this
> time and I did not set up any periodic recording into a
> file.
> 
> That is why I've not reported on the RAM+SWAP total
> this time. It will have to be another experiment
> some other time.
> 
> [I do wish FreeBSD had a way of reporting peak swap-space
> usage.]

I've also tried without WITH_DEBUG= and now. . .


# pkg delete llvm40
Checking integrity... done (0 conflicting)
Deinstallation has been requested for the following 1 packages (of 0 packages in the universe):

Installed packages to be REMOVED:
	llvm40-4.0.0

Number of packages to be removed: 1

The operation will free 1 GiB.

Proceed with deinstalling packages? [y/N]: n


# du -sg /usr/obj/portswork/usr/ports/devel/llvm40/
5	/usr/obj/portswork/usr/ports/devel/llvm40/


So the alternatives (with everything built each time):

(5+1)    GiBytes ==   6 GiBytes. (without WITH_DEBUG=)
vs.
(102+42) GiBytes == 146 GiBytes. (WITH_DEBUG= but the adjusted llvm40/Makefiele)
vs.
(118+49) GiBytes == 167 GiBytes. (WITH_DEBUG= with Makefile adjustment)


I'll likely end up having /etc/make.conf contain
something like the following for most or all of
my FreeBSD environments:

#
# From a local /usr/ports/Mk/bsd.port.mk extension:
ALLOW_OPTIMIZATIONS_FOR_WITH_DEBUG=
#
.if ${.CURDIR:M*/devel/llvm*}
#WITH_DEBUG=
.else
WITH_DEBUG=
.endif
WITH_DEBUG_FILES=


Along with using:

# svnlite diff /usr/ports/Mk/
Index: /usr/ports/Mk/bsd.port.mk
===================================================================

--- /usr/ports/Mk/bsd.port.mk	(revision 436747)
+++ /usr/ports/Mk/bsd.port.mk	(working copy)
@@ -1646,7 +1646,11 @@
 STRIP_CMD=	${TRUE}
 .endif
 DEBUG_FLAGS?=	-g
+.if defined(ALLOW_OPTIMIZATIONS_FOR_WITH_DEBUG)
+CFLAGS:=		${CFLAGS} ${DEBUG_FLAGS}
+.else
 CFLAGS:=		${CFLAGS:N-O*:N-fno-strict*} ${DEBUG_FLAGS}
+.endif
 .if defined(INSTALL_TARGET)
 INSTALL_TARGET:=	${INSTALL_TARGET:S/^install-strip$/install/g}
 .endif


[Although apparently the ALLOW_OPTIMIZATIONS_FOR_WITH_DEBUG
use has no effect for devel/llvm40 .]

===
Mark Millard
markmi at dsl-only.net

