Cannot unmount nullfs in current

Fri Apr 19 02:04:44 UTC 2013

Hi,

I am trying to build some software which uses
nanobsd, and mounts/unmounts many nullfs mounts
while it runs.  I am hitting failures where
I cannot unmount nullfs file systems.  I cannot figure out why.

Here is more info.

SYSTEM
======
I am running amd64, current build at this revision:

10.0-CURRENT FreeBSD 10.0-CURRENT #0 r249181: Sat Apr  6 03:07:32 UTC 2013
amd64

STEPS TO REPRODUCE
===============

(1)  Create a directory, /opt2/branches.  Make sure that /opt2/branches
     is on ZFS

(2)
       mkdir -p /opt2/branches/freenas
       mkdir -p /opt2/branches/freenas-cache

(3)

git clone git://github.com/freenas/freenas.git /opt2/branches/freenas
git clone git://github.com/freenas/ports.git/opt2/branches/freenas-cache/ports
git clone git://github.com/trueos/trueos.git/opt2/branches/freenas-cache/trueos

(4)  sudo to root

(5)  cd /opt2/branches/freenas

(6)
script build.log env GIT_REPO=/opt2/branches/freenas-cache/trueos \
        GIT_PORTS_REPO=/opt2/branches/freenas-cache/ports \
        sh build/do_build.sh

The build cranks for a while, and then I get this error:

00:02:37 ### log:
/opt2/branches/freenas/os-base/amd64/_.cust.add_pkg_archivers_lzo2
do_build.sh: ERROR: FreeNAS /opt2/branches/freenas/nanobsd/os-base build
FAILED; please check above log for more details

If I look in .cust.add_pkg_archivers_lzo2, I see this error:

+ umount /opt2/branches/freenas/os-base/amd64/_.w/usr/ports/distfiles
umount: unmount of
/opt2/branches/freenas/os-base/amd64/_.w/usr/ports/distfiles failed: Device
busy

If I try to do this manually:

# umount /opt2/branches/freenas/os-base/amd64/_.w/usr/ports/distfiles
umount: unmount of
/opt2/branches/freenas/os-base/amd64/_.w/usr/ports/distfiles failed: Device
busy

I can't figure out why this mount is busy.
If I do:

umount -f /opt2/branches/freenas/os-base/amd64/_.w/usr/ports/distfiles

it unmounts, but I don't like using the '-f' flag to force the unmount.

Any ideas?  I am attaching some of my logs.

--
Craig
-------------- next part --------------
Script started on Thu Apr 18 14:29:00 2013
command: env GIT_REPO=/opt2/branches/freenas-cache/trueos GIT_PORTS_REPO=/opt2/branches/freenas-cache/ports sh build/do_build.sh
Using local mirror in /opt2/branches/freenas-cache/trueos.
Using local git ports mirror in /opt2/branches/freenas-cache/ports
Sourcing /opt2/branches/freenas/nanobsd/os-base
Package is not yet built: ftp_wget
Package is not yet built: benchmarks_iozone
Package is not yet built: benchmarks_iperf
Package is not yet built: benchmarks_netperf
Package is not yet built: benchmarks_xdd
Package is not yet built: security_sudo
Package is not yet built: sysutils_ipmitool
Package is not yet built: www_py-django-json-rpc
Package is not yet built: devel_py-mimeparse
Package is not yet built: devel_py-six
Package is not yet built: devel_py-dateutil
Package is not yet built: devel_py-rose
Package is not yet built: www_py-django-tastypie
Package is not yet built: devel_py-daemon
Package is not yet built: devel_py-polib
Package is not yet built: devel_py-ujson
Package is not yet built: devel_py-simplejson
Package is not yet built: sysutils_bsdstats
Package is not yet built: www_wgetpaste
Package is not yet built: devel_py-greenlet
Package is not yet built: net_py-eventlet
Package is not yet built: graphics_jpeg
Package is not yet built: security_ca_root_nss
Package is not yet built: ftp_curl
Package is not yet built: devel_apr1
Package is not yet built: www_neon29
Package is not yet built: devel_subversion
Package is not yet built: editors_vim-lite
Package is not yet built: misc_py-pexpect
Package is not yet built: devel_ipython
Package is not yet built: devel_p5-Term-ReadKey
Package is not yet built: devel_p5-subversion
Package is not yet built: mail_p5-Net-SMTP-SSL
Package is not yet built: lang_p5-Error
Package is not yet built: devel_git
Package is not yet built: devel_ctags
Automatically building a * * F A T * * image so we can build ports
00:00:00 # NanoBSD image FreeNAS-9.1.0-ALPHA-4dd41d9_dirty-x64 build starting
00:00:00 ## Skipping buildworld (as instructed)
00:00:00 ## Skipping buildkernel (as instructed)
00:00:00 ## Clean and create world directory (/opt2/branches/freenas/os-base/amd64/_.w)
00:00:25 ## Construct install make.conf (/opt2/branches/freenas/os-base/amd64/make.conf.install)
00:00:25 ## installworld
00:00:25 ### log: /opt2/branches/freenas/os-base/amd64/_.iw
00:02:29 ## install /etc
00:02:29 ### log: /opt2/branches/freenas/os-base/amd64/_.etc
00:02:30 ## configure nanobsd /etc
00:02:30 ## install kernel (/opt2/branches/freenas/nanobsd/FREENAS.amd64)
00:02:30 ### log: /opt2/branches/freenas/os-base/amd64/_.ik
00:02:37 ## run customize scripts
00:02:37 ## [1/184] customize "clean_packages"
00:02:37 ### log: /opt2/branches/freenas/os-base/amd64/_.cust.clean_packages
00:02:37 ## Clean and create world directory (/opt2/branches/freenas/os-base/amd64/_.w/usr/local)
00:02:37 ## [2/184] customize "cust_install_files"
00:02:37 ### log: /opt2/branches/freenas/os-base/amd64/_.cust.cust_install_files
00:02:37 ## [3/184] customize "cust_install_machine_files"
00:02:37 ### log: /opt2/branches/freenas/os-base/amd64/_.cust.cust_install_machine_files
00:02:37 ## [4/184] customize "fetch_background"
00:02:37 ### log: /opt2/branches/freenas/os-base/amd64/_.cust.fetch_background
00:02:37 ## fetch ports in background
00:02:37 ## [5/184] customize "generate_avatar_conf"
00:02:37 ### log: /opt2/branches/freenas/os-base/amd64/_.cust.generate_avatar_conf
00:02:37 ## [6/184] customize "add_pkg_archivers_lzo2"
00:02:37 ### log: /opt2/branches/freenas/os-base/amd64/_.cust.add_pkg_archivers_lzo2
do_build.sh: ERROR: FreeNAS /opt2/branches/freenas/nanobsd/os-base build FAILED; please check above log for more details

Script done on Thu Apr 18 14:32:21 2013
-------------- next part --------------
+ add_pkg_archivers_lzo2
+ do_add_pkg lzo2-2.06
+ mkdir -p /opt2/branches/freenas/os-base/amd64/ports/distfiles
+ mkdir -p /opt2/branches/freenas/os-base/amd64/ports/packages
+ mkdir -p /opt2/branches/freenas/os-base/amd64/_.w/usr/ports/packages/All
+ mkdir -p /opt2/branches/freenas/os-base/amd64/_.w/usr/ports/distfiles
+ mount -t nullfs -o noatime /opt2/branches/freenas/os-base/amd64/ports/packages /opt2/branches/freenas/os-base/amd64/_.w/usr/ports/packages
+ mount -t nullfs -o noatime /opt2/branches/freenas/os-base/amd64/ports/distfiles /opt2/branches/freenas/os-base/amd64/_.w/usr/ports/distfiles
+ CR 'cd /usr/ports/packages/All;pkg_add -F lzo2-2.06.tbz'
+ mount -t devfs none /opt2/branches/freenas/os-base/amd64/_.w/dev
+ fake_target_host chroot /opt2/branches/freenas/os-base/amd64/_.w /bin/sh -exc 'cd /usr/ports/packages/All;pkg_add -F lzo2-2.06.tbz'
+ local newvers revision branch
+ newvers=/opt2/branches/freenas/FreeBSD/src/sys/conf/newvers.sh
+ grep -m 1 REVISION= /opt2/branches/freenas/FreeBSD/src/sys/conf/newvers.sh
+ cut -f2 -d '"'
+ revision=9.1
+ grep -m 1 BRANCH= /opt2/branches/freenas/FreeBSD/src/sys/conf/newvers.sh
+ cut -f2 -d '"'
+ branch=STABLE
+ set -e
+ env
SUDO_COMMAND=/usr/local/bin/bash
NANO_SRC=/opt2/branches/freenas/FreeBSD/src
NANO_MODULES=cxgb ext2fs geom ipmi iscsi pf pflog smbfs libiconv libmchain syscons udf zfs opensolaris usb/xhci
NANO_LABEL=FreeNAS
FORCE_FBSD_ONLY=1
NANO_MEDIASIZE=3906250
NANO_NEWFS=-b 4096 -f 512 -i 8192 -O1 -U
LOGNAME=root
NANO_TOOLS=/opt2/branches/freenas/build/nanobsd
NANO_MAKE_CONF_BUILD=/opt2/branches/freenas/os-base/amd64/make.conf.build
NANO_BOOTLOADER=boot/boot0
NANO_CUSTOMIZE= clean_packages cust_install_files cust_install_machine_files fetch_background generate_avatar_conf add_pkg_archivers_lzo2 add_pkg_security_easy-rsa add_pkg_security_openvpn add_pkg_converters_libiconv add_pkg_lang_perl5.14 add_pkg_converters_iconv add_pkg_devel_gettext add_pkg_devel_pkgconf add_pkg_devel_libpthread-stubs add_pkg_devel_cdialog add_pkg_dns_inadyn-mt add_pkg_devel_pcre add_pkg_editors_nano add_pkg_ftp_proftpd add_pkg_net-mgmt_bsnmp-ucd add_pkg_net-mgmt_bsnmptools add_pkg_net-mgmt_clog add_pkg_net-mgmt_sipcalc add_pkg_net_istgt add_pkg_net_vblade add_pkg_security_libgpg-error add_pkg_security_libgcrypt add_pkg_databases_db46 add_pkg_security_cyrus-sasl2 add_pkg_net_openldap24-sasl-client add_pkg_net_nss_ldap add_pkg_net_rsync add_pkg_lang_python27 add_pkg_dns_py-dnspython add_pkg_devel_talloc add_pkg_devel_libexecinfo add_pkg_devel_popt add_pkg_databases_tdb add_pkg_sysutils_libsunacl add_pkg_net_samba36 add_pkg_security_pam_ldap add_pkg_security_pam_mkhomedir add_pkg_shells_bash add_pkg_shells_scponly add_pkg_sysutils_e2fsprogs add_pkg_sysutils_fusefs-kmod add_pkg_sysutils_fusefs-libs add_pkg_devel_libublio add_pkg_sysutils_fusefs-ntfs add_pkg_sysutils_smartmontools add_pkg_devel_glib20 add_pkg_net_libdnet add_pkg_emulators_open-vm-tools-nox11 add_pkg_databases_sqlite3 add_pkg_databases_py-sqlite3 add_pkg_databases_py-bsddb3 add_pkg_devel_py-distribute add_pkg_www_py-flup add_pkg_www_py-django add_pkg_www_py-dojango add_pkg_www_dojo add_pkg_databases_py-south add_pkg_devel_py-asn1 add_pkg_devel_py-asn1-modules add_pkg_www_nginx add_pkg_net-mgmt_net-snmp add_pkg_sysutils_nut add_pkg_textproc_libxml2 add_pkg_textproc_py-libxml2 add_pkg_textproc_expat2 add_pkg_devel_gamin add_pkg_devel_gio-fam-backend add_pkg_devel_m4 add_pkg_devel_bison add_pkg_devel_libffi add_pkg_misc_gnomehier add_pkg_devel_gobject-introspection add_pkg_www_py-httplib2 add_pkg_net_py-oauth2 add_pkg_sysutils_jailme add_pkg_graphics_png add_pkg_devel_libstatgrab add_pkg_devel_libltdl add_pkg_print_freetype2 add_pkg_x11_xproto add_pkg_x11-fonts_fontconfig add_pkg_x11-fonts_libfontenc add_pkg_x11-fonts_mkfontscale add_pkg_x11-fonts_mkfontdir add_pkg_x11-fonts_font-bh-ttf add_pkg_x11-fonts_font-misc-meltho add_pkg_x11-fonts_font-misc-ethiopic add_pkg_x11-fonts_bitstream-vera add_pkg_x11-fonts_font-util add_pkg_x11-fonts_encodings add_pkg_x11-fonts_xorg-fonts-truetype add_pkg_x11_pixman add_pkg_graphics_cairo add_pkg_x11-toolkits_pango add_pkg_databases_rrdtool add_pkg_net_liboping add_pkg_net-mgmt_collectd5 add_pkg_devel_py-ipaddr add_pkg_converters_base64 add_pkg_emulators_mtools add_pkg_sysutils_arcconf add_pkg_sysutils_tw_cli add_pkg_sysutils_megacli add_pkg_sysutils_areca-cli add_pkg_net_py-ldap2 add_pkg_sysutils_ataidle add_pkg_sysutils_gnome_subr add_pkg_devel_dbus add_pkg_devel_dbus-glib add_pkg_devel_libdaemon add_pkg_databases_gdbm add_pkg_net_avahi-app add_pkg_net_avahi-libdns add_pkg_textproc_py-xml add_pkg_sysutils_throttle add_pkg_sysutils_dmidecode add_pkg_sysutils_graid5 add_pkg_devel_libevent add_pkg_sysutils_tmux add_pkg_net_netatalk add_pkg_dns_libidn add_port_ftp_wget add_port_benchmarks_iozone add_port_benchmarks_iperf add_port_benchmarks_netperf add_port_benchmarks_xdd add_port_security_sudo add_port_sysutils_ipmitool add_port_www_py-django-json-rpc add_port_devel_py-mimeparse add_port_devel_py-six add_port_devel_py-dateutil add_port_devel_py-rose add_port_www_py-django-tastypie add_port_devel_py-daemon add_port_devel_py-polib add_port_devel_py-ujson add_port_devel_py-simplejson add_port_sysutils_bsdstats add_port_www_wgetpaste add_port_devel_py-greenlet add_port_net_py-eventlet add_port_graphics_jpeg add_port_security_ca_root_nss add_port_ftp_curl add_port_devel_apr1 add_port_www_neon29 add_port_devel_subversion add_port_editors_vim-lite add_port_misc_py-pexpect add_port_devel_ipython add_port_devel_p5-Term-ReadKey add_port_devel_p5-subversion add_port_mail_p5-Net-SMTP-SSL add_port_lang_p5-Error add_port_devel_git add_port_devel_ctags hack_nsswitch_conf add_gui add_autotune arch_specific_ko add_pbi_manager add_warden add_pcbsd_libsh add_pbi_wrapper move_data add_data_to_fstab afpd_conf_symlink select_httpd static_dialog remove_patch_divots configure_mnt_md shrink_md_fbsize save_build unmute_console_logging remove_var_db_pkg fix_fuse_module_location fix_easy_install_pth create_var_home_symlink
AVATAR_ROOT=/opt2/branches/freenas
NANO_XZ=xz
PBI_DELETE_BUILD=0
MAKEOBJDIRPREFIX=/opt2/branches/freenas/os-base/amd64
NANO_PMAKE=make -j 5
SCRIPT=build.log
NANO_ARCH=amd64
MAIL=/var/mail/root
NANO_LOCAL_DIRS=pbi-wrapper
NANO_CONFSIZE=2048
NANO_HEADS=16
MAKE_JOBS=5
PATH=/opt2/branches/freenas/sbin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/opt/home/rodrigc/bin:/usr/local/Acrobat5/bin:/usr/local/bin:/usr/local/sbin:/volume/buildtools/bin:/volume/labtools/bin:/volume/current/sw-projects/review-tracker/scripts
PBI_DELETE_BUILDSRC=0
NANO_IMAGES=2
NANO_CFG_BASE=/opt2/branches/freenas/nanobsd
NANO_MAKE_CONF_INSTALL=/opt2/branches/freenas/os-base/amd64/make.conf.install
SUDO_GID=1001
OLDPWD=/opt2/branches/freenas/FreeBSD/ports/devel/git
PBI_APPDIR=/opt2/branches/freenas/pbi
NANO_DATASIZE=40960
PWD=/opt2/branches/freenas/FreeBSD/ports/devel/ctags
_=/usr/bin/script
REVISION=4dd41d9_dirty
NANO_WORLDDIR=/opt2/branches/freenas/os-base/amd64/_.w
FREENAS_ARCH=amd64
PBI_BUILDSRC=/opt2/branches/freenas/pbi/src
NANO_ARCH_HUMANIZED=x64
TERM=xterm
NANO_NAME=FreeNAS-9.1.0-ALPHA-4dd41d9_dirty-x64
NANO_IMGNAME=FreeNAS-9.1.0-ALPHA-4dd41d9_dirty-x64
NANO_MAKEFS=makefs -B big 	-o bsize=4096,fsize=512,density=8192,optimization=space
NANO_CODESIZE=0
USER=root
HOME=/root
PBI_BUILDTARGET=/opt2/branches/freenas/pbi/amd64/target
NANO_DRIVE=ufs/FreeNAS
NANO_ARGS=
NANO_KERNEL=/opt2/branches/freenas/nanobsd/FREENAS.amd64
NANO_BOOT0CFG=-o packet -s 1 -m 3 -t 18
PS1=[\u@\h \W]% 
GIT_REPO=/opt2/branches/freenas-cache/trueos
SHELL=/usr/local/bin/bash
SVNREVISION=4dd41d9_dirty
NANO_SECTS=63
SUDO_USER=rodrigc
PBI_PKGDIR=/opt2/branches/freenas/pbi/amd64/pkgs
SUDO_UID=1001
USERNAME=root
PBI_OSREL=9.0-RELEASE
GIT_PORTS_REPO=/opt2/branches/freenas-cache/ports
PBI_BUILDDIR=/opt2/branches/freenas/pbi/amd64
PBI_OSARCH=amd64
AVATAR_COMPONENT=os-base
PBI_BINDIR=/opt2/branches/freenas/sbin
PYTHON_DEFAULT_VERSION=python2.7
NANO_OBJ=/opt2/branches/freenas/os-base/amd64
PBI_DELETE_BUILDTARGET=0
SHLVL=1
+ export UNAME_m=amd64
+ export UNAME_p=amd64
+ export UNAME_r=9.1-STABLE
+ uname -v
+ uname -p
+ uname -r
+ sed -e s/amd64/amd64/ -e s/9.1-STABLE/9.1-STABLE/g
+ export 'UNAME_v=FreeBSD 10.0-CURRENT #0 r249181: Sat Apr  6 03:07:32 UTC 2013     rodrigc at dibbler.crodrigues.org:/usr/obj/opt2/branches/head/sys/GENERIC '
+ awk '/\#define.*__FreeBSD_version/ { print $3 }' /opt2/branches/freenas/FreeBSD/src/sys/sys/param.h
+ export OSVERSION=901504
+ export PATH=/opt2/branches/freenas/sbin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin
+ chroot /opt2/branches/freenas/os-base/amd64/_.w /bin/sh -exc 'cd /usr/ports/packages/All;pkg_add -F lzo2-2.06.tbz'
+ cd /usr/ports/packages/All
+ pkg_add -F lzo2-2.06.tbz
+ umount /opt2/branches/freenas/os-base/amd64/_.w/dev
+ umount /opt2/branches/freenas/os-base/amd64/_.w/usr/ports/distfiles
umount: unmount of /opt2/branches/freenas/os-base/amd64/_.w/usr/ports/distfiles failed: Device busy