git: cb1d034ab1 - main - docbuild: add scripts used to build the PDF versions of the Handbook

Thu Dec 31 12:29:18 UTC 2020

Hey,

I've got a couple of minor suggestions and questions regarding this patch.

On 12/30/20 5:53 PM, Glen Barber wrote:
> The branch main has been updated by gjb:
>
> URL: https://cgit.FreeBSD.org/doc/commit/?id=cb1d034ab1e773758d20238be196aeeb1be68b18
>
> commit cb1d034ab1e773758d20238be196aeeb1be68b18
> Author:     Glen Barber <gjb at FreeBSD.org>
> AuthorDate: 2020-12-30 16:40:07 +0000
> Commit:     Glen Barber <gjb at FreeBSD.org>
> CommitDate: 2020-12-30 16:40:07 +0000
>
>      docbuild: add scripts used to build the PDF versions of the Handbook

> diff --git a/share/tools/dobuild.sh b/share/tools/dobuild.sh
> new file mode 100755
> index 0000000000..a2ae5c092d
> --- /dev/null
> +++ b/share/tools/dobuild.sh
> @@ -0,0 +1,117 @@
> +#!/bin/sh
> +# Copyright (c) 2001 Wolfram Schneider <wosch at FreeBSD.org>
> +# Copyright (c) 2001 Dima Dorfman <dd at FreeBSD.org>
> +# Copyright (c) 2004 Simon L. Nielsen <simon at FreeBSD.org>
> +#
> +# Build the FreeBSD docs from the CVS^WSVN repository.
> +#
> +#
> +# Major variables:
> +#
> +#	PATH		- The search path as interpreted by the shell.
> +#	CVSROOT		- Path to the FreeBSD CVS repository (obsolete)
> +#	DOCSVN		- Url to the FreeBSD doc SVN repository.
> +#	BUILDDIR	- Where the checked out copies of the files are stored.
> +#	DESTDIR		- Where the rendered copies should wind up.
> +#	BUILDARGS	- Arguments to pass to make(1) when {build,install}ing.
> +#	INSTARGS	- Arguments to pass to make(1) when installing.
> +#	WEBMAILTO	- Address to send mail to if the build fails.
> +#
> +#	subtrees	- List of directores in $BUILDDIR which are from SCM.
> +#
> +# Variables which are in uppercase are derived from the environment
> +# unless they don't exist, in which case a value suitable for
> +# www.FreeBSD.org is used.  Variables in lowercase can't be safely
> +# changed without editing other parts of the script; thus, their value
> +# in the environment is ignored.
> +#
> +# Exit codes:
> +#
> +#	0	- success
> +#	1	- unknown failure
> +#	2	- failure in CVS^SVN operations
> +#	3	- failure in make operations
> +#
> +# $FreeBSD: www/tools/webupdate,v 1.10 2003/11/18 11:00:54 kuriyama Exp $
> +#
> +
> +#
> +# WARNING!  This script depend on dobuild_wrap.sh setting the
> +# configuration variables in the enviroment.
> +#
> +if [ -z "${DOBUILDWRAP}" ]; then
> +	echo "Error: This script should only be called from dobuild_wrap.sh!"
> +	exit 1
> +fi
> +
> +# Only install some compression types
> +INSTALL_COMPRESSED=${COMPTYPES}; export INSTALL_COMPRESSED
> +
> +# Abort on all errors
> +set -e
> +
> +umask 002
> +cd $BUILDDIR || exit 1
> +
> +# Remove the old copies.
> +if [ -z "${NOCLEAN}" ]; then
> +	echo "===> Removing old stuff"
> +	rm -Rf doc out 2>/dev/null || true
> +	chflags -R noschg doc out 2>/dev/null || true
> +	rm -Rf doc out 2>/dev/null
> +
> +	echo "===> Check out the new doc"
> +	#cvs -qR checkout -P doc || exit 2
> +	svn checkout ${DOCSVN} doc || exit 2
> +fi
> +
> +cd $BUILDDIR/doc || exit 1
> +mkdir -p $DOCDIR
> +mkdir -p $DOCDIR/packages
> +rm -f ${BUILDDIR}/packages # TMP
> +ln -s ${DOCDIR}/packages ${BUILDDIR}/doc/packages
> +
> +if [ -d ${PATCHDIR} -a -n "$(find ${PATCHDIR} -name \*.patch)" ]; then
> +	echo "===> Patching bugs in doc tree"
> +	for p in `echo ${PATCHDIR}/*.patch`; do

Could this be just

```
for p in ${PATCHDIR}/*.patch; do
```

instead?

> +		patch < ${p}
> +	done
> +fi
> +
> +echo "===> Building doc"
> +time make ${BUILDARGS} all || exit 3
> +
> +echo "===> Installing doc package to temp roots"
> +time make ${INSTARGS} install || exit 3
> +
> +#echo "===> Building doc packages"
> +#time make ${BUILDARGS} package || exit 3
> +
> +# XXX TODO, check for correct string when tex run out of resources
> +#echo "===> Testing for broken tex output"
> +#grep 'hash' $LOGFILE > /dev/null && \
> +#    (echo "Error: Possible TeX out of resources."; false)
> +
> +#OKFILENAME
> +
> +# build a string to use with find(1) to identify
> +# uncompressed files to be copied
> +UNCOMP_FIND_STR=""
> +for fmt in ${UNCOMPRESSED_FORMATS}; do
> +  UNCOMP_FIND_STR="${UNCOMP_FIND_STR}( -name article.${fmt} -or -name book.${fmt} ) -or "
> +done
> +# add outer parens, remove the final "-or "
> +UNCOMP_FIND_STR="( ${UNCOMP_FIND_STR%-or } )"
> +
> +if [ -n "${UNCOMP_FIND_STR}" ]; then
> +  echo "copying uncompressed formats: ${UNCOMPRESSED_FORMATS}"
> +  cd ${BUILDDIR}/doc
> +  find -X ??_* ${UNCOMP_FIND_STR} -print0 \
> +  | xargs -0 -I % install -v -C -o docbuild -g docbuild -m 444 % ${DOCDIR}/%
> +fi
> +
> +rm -rf ${OUTDIR}/*
> +mkdir -p ${OUTDIR}
> +cp -Rp ${DOCDIR}/* ${OUTDIR}
> +
> +exit 0
> diff --git a/share/tools/dobuild_wrap.sh b/share/tools/dobuild_wrap.sh
> new file mode 100755
> index 0000000000..d0bdf3e246
> --- /dev/null
> +++ b/share/tools/dobuild_wrap.sh
> @@ -0,0 +1,135 @@
> +#!/bin/sh
> +#
> +# Copyright (c) 2004 Simon L. Nielsen <simon at FreeBSD.org>
> +# All rights reserved.
> +#
> +# Redistribution and use in source and binary forms, with or without
> +# modification, are permitted provided that the following conditions
> +# are met:
> +# 1. Redistributions of source code must retain the above copyright
> +#    notice, this list of conditions and the following disclaimer.
> +# 2. Redistributions in binary form must reproduce the above copyright
> +#    notice, this list of conditions and the following disclaimer in the
> +#    documentation and/or other materials provided with the distribution.
> +#
> +# THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
> +# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
> +# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
> +# ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
> +# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
> +# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
> +# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
> +# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
> +# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
> +# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
> +# SUCH DAMAGE.
> +#
> +# $FreeBSD$
> +#
> +
> +# Wrapper around doc build.  Simplifies main build script.
> +
> +#
> +# Default configuration.
> +#
> +DEFAULT_PATH=/bin:/usr/bin:/usr/local/bin
> +DEFAULT_BUILDROOT=/local0/docbuild
> +DEFAULT_CVSROOT=/home/dcvs
> +DEFAULT_DOCSVN="svn://svn.freebsd.org/doc/head"
> +DEFAULT_BUILDARGS="NO_JPMAN=yes GEN_INDEX=yes"
> +DEFAULT_INSTARGS="NO_JPMAN=yes GEN_INDEX=yes INSTALL_ONLY_COMPRESSED=yes"
> +# Must keep space between addresses.
> +DEFAULT_WEBMAILTO="doceng at FreeBSD.org, wblock at wonkity.com"
> +#DEFAULT_FORMATS="html html.tar html-split html-split.tar txt ps pdf rtf pdb"
> +#DEFAULT_FORMATS="html html.tar html-split html-split.tar txt ps pdf epub"
> +DEFAULT_FORMATS="html html.tar html-split html-split.tar txt pdf epub"
> +DEFAULT_UNCOMPRESSED_FORMATS="pdf epub"
> +DEFAULT_COMPTYPES="bz2 zip"
> +
> +#
> +# Variable setup.
> +#
> +BUILDROOT=${BUILDROOT:-${DEFAULT_BUILDROOT}}; export BUILDROOT
> +PATH="${PATH}:${DEFAULT_PATH}"; export PATH
> +CVSROOT=${CVSROOT:-${DEFAULT_CVSROOT}}; export CVSROOT
> +DOCSVN=${DOCSVN:-${DEFAULT_DOCSVN}}; export DOCSVN
> +BUILDDIR=${BUILDDIR:-${BUILDROOT}/build}; export BUILDDIR
> +DOCDIR=${DOCDIR:-${BUILDROOT}/build/out}; export DOCDIR
> +FORMATS=${FORMATS:-${DEFAULT_FORMATS}}; export FORMATS
> +UNCOMPRESSED_FORMATS=${UNCOMPRESSED_FORMATS:-${DEFAULT_UNCOMPRESSED_FORMATS}}; export UNCOMPRESSED_FORMATS
> +COMPTYPES=${COMPTYPES:-${DEFAULT_COMPTYPES}}; export COMPTYPES
> +BUILDARGS=${BUILDARGS:-${DEFAULT_BUILDARGS}}; export BUILDARGS
> +INSTARGS=${INSTARGS:-${DEFAULT_INSTARGS}}; export INSTARGS
> +WEBMAILTO=${WEBMAILTO:-${DEFAULT_WEBMAILTO}}; export WEBMAILTO
> +PATCHDIR=${PATCHDIR:-${BUILDROOT}/patches}; export PATCHDIR
> +OUTDIR=${OUTDIR:-${BUILDROOT}/docout}; export OUTDIR
> +LOGROOT=${LOGBUILDROOT:-${BUILDROOT}/logs}; export LOGROOT
> +LOGDIR=${LOGROOT}/$(date +%Y%m%d%H%M%S); export LOGDIR
> +LOGFILE=${LOGDIR}/buildlog; export LOGFILE
> +LOCKFILE=${LOCKFILE:-${BUILDROOT}/lock}; export LOCKFILE
> +STATUSFILE=${LOGDIR}/status.xml; export STATUSFILE
> +BUILDSFILE=${LOGROOT}/builds.xml; export BUILDSFILE
> +
> +DOBUILDWRAP="yes"; export DOBUILDWRAP
> +
> +# Run the actual script inside a lock so only one instance can run at
> +# the time.
> +lockf -t 0 ${LOCKFILE} /bin/sh << 'E*O*F'
> +	if [ ! -d ${LOGDIR} ]; then
> +		mkdir -p ${LOGDIR}
> +	fi
> +	LOGFILE_REL=${LOGFILE##${LOGROOT}/}
> +	echo "<build>"						 > ${STATUSFILE}
> +	echo " <buildhost>$(hostname)</buildhost>"		>> ${STATUSFILE}
> +	echo " <starttime>$(date +%Y%m%d%H%M%S)</starttime>"	>> ${STATUSFILE}
> +	echo " <logfile>${LOGFILE_REL}</logfile>"		>> ${STATUSFILE}
> +	echo " <status>running</status>"			>> ${STATUSFILE}
> +	echo "</build>"						>> ${STATUSFILE}
> +
> +	echo '<builds>'						 > ${BUILDSFILE}
> +	find ${LOGROOT} -name status.xml | xargs cat		>> ${BUILDSFILE}
> +	echo '</builds>'					>> ${BUILDSFILE}
> +
> +	# Do the actual build.
> +	starttime=$(date +%s)
> +	sh -x ${BUILDROOT}/dobuild.sh > ${LOGFILE} 2>&1
> +	ret=$?
> +	touch ${LOGFILE}
> +	endtime=$(date +%s)
> +	runtime=$((${endtime} - ${starttime}))
> +
> +	# Remove the status and "footer" of ${STATUSFILE}.
> +	keeplines=$(($(wc -l < ${STATUSFILE}) - 2))
> +	mv ${STATUSFILE} ${STATUSFILE}.tmp
> +	head -n ${keeplines} ${STATUSFILE}.tmp > ${STATUSFILE}
> +	rm -f ${STATUSFILE}.tmp
> +
> +	echo " <endtime>$(date +%Y%m%d%H%M%S)</endtime>"	>> ${STATUSFILE}
> +	echo " <runtime>${runtime}</runtime>"			>> ${STATUSFILE}
> +	echo " <retcode>${ret}</retcode>"			>> ${STATUSFILE}
> +	if [ ${ret} -eq 0 ]; then
> +		echo " <status>completed</status>"		>> ${STATUSFILE}
> +	else
> +		echo " <status>failed</status>"			>> ${STATUSFILE}
> +	fi
> +
> +	if [ ${ret} -ne 0 ]; then
> +		tail -100 ${LOGFILE} | \
> +			mail -s "FreeBSD FTP doc build failed" ${WEBMAILTO}
> +	fi
> +
> +	if egrep '^! TeX capacity exceeded' ${LOGFILE} > /dev/null; then
> +		egrep '^! TeX capacity exceeded' ${LOGFILE} | \
> +			mail -s "FreeBSD FTP doc build warning" ${WEBMAILTO}
> +		echo " <warning>Possibly out of TeX resources.</warning>" \
> +			>> ${STATUSFILE}
> +	fi
> +	echo "</build>"						>> ${STATUSFILE}
> +
> +	echo '<builds>'						 > ${BUILDSFILE}
Is this ">" instead of ">>" on purpose?
> +	find ${LOGROOT} -name status.xml | xargs cat		>> ${BUILDSFILE}
"-exec cat +" might do as well.
> +	echo '</builds>'					>> ${BUILDSFILE}
> +E*O*F
> +
> +
> +exit 0