bsdtar and packages vs. unionfs (was: Re: Cannot package converters/libiconv inside clean chroot)

[Ulrich Spoerlein]

Tim Kientzle wrote:
> >There are at least two issues here, one is pkg_add refusing a valid
> >(AFAICS) tbz file, the other is bsdtar(1) choosing a different tar
> >format based on unionfs(!).
> 
> Those two symlink entries have an "opaque" file flag.
> This explains the format change (bsdtar uses the extended
> pax format when it sees a file with flags set, since
> ustar can't store those).
> 
> I would guess that pkg_add is invoking bsdtar with
> -p (restore permissions), bsdtar is restoring the
> 'opaque' flag, and then pkg_add is tripping over
> those symlinks for some reason when it tries to
> move them.

/usr/src/usr.sbin/pkg_install/add/extract.c:37:   strcat(where_args, "|/usr/bin/tar --unlink -xpf - -C "); \

> To test this hypothesis, try stripping those flags
> with:
> 
>    tar -cjf new-libiconv-1.9.2_2.tbz --format=ustar @libiconv-1.9.2_2.tbz
> 
> The "new-libiconv" tarfile should be identical except
> it won't have the file flags stored.  If pkg_add
> likes new-libiconv but not libiconv, then it must
> be those opaque flags.

Yes, using the trick above, pkg_add no longer complains. I can use this
as a temporary workaround. I think the real problem lies with bsdtar(1),
though. See below.

> I don't know if this is a bug in unionfs, in pkg_add,
> or in bsdtar.  I think we need someone who understands
> the 'opaque' flag to chime in here.

I was certain, that I tried to extract the "broken" package with the
exact same flags as pkg_add uses (--unlink -xpf) but it looks like I
messed something up, as _now_ I do see the same errors with bsdtar
itself.

roadrunner# rm -rf foo ; mkdir foo ; tar --unlink -xpvf libiconv-1.9.2_2.tbz -C foo
x +CONTENTS
x +COMMENT
x +DESC
x +MTREE_DIRS
x man/man1/iconv.1.gz
x man/man3/iconv.3.gz
x man/man3/iconv_open.3.gz
x man/man3/iconv_close.3.gz
x bin/iconv
x include/iconv.h
x include/libcharset.h
x include/localcharset.h
x lib/libcharset.a
x lib/libcharset.la
x lib/libcharset.so: Couldn't stat file: No such file or directory
x lib/libcharset.so.1
x lib/libiconv.a
x lib/libiconv.la
x lib/libiconv.so: Couldn't stat file: No such file or directory
x lib/libiconv.so.3
x libdata/charset.alias
x share/doc/libiconv/iconv.1.html
x share/doc/libiconv/iconv.3.html
x share/doc/libiconv/iconv_close.3.html
x share/doc/libiconv/iconv_open.3.html
roadrunner# echo $?
1
roadrunner# find foo -exec ls -dlo {} \+
drwxr-xr-x  8 root  wheel  opaque     512 Apr  9 10:34 foo
-rw-r--r--  1 root  wheel  -           35 Apr  9 09:47 foo/+COMMENT
-rw-r--r--  1 root  wheel  -         2427 Apr  9 09:47 foo/+CONTENTS
-rw-r--r--  1 root  wheel  -          676 Apr  9 09:47 foo/+DESC
-rwxr-xr-x  1 root  wheel  -        15305 Apr  9 09:47 foo/+MTREE_DIRS
drwxr-xr-x  2 root  wheel  opaque     512 Apr  9 10:34 foo/bin
-r-xr-xr-x  1 root  wheel  -         7724 Apr  9 09:47 foo/bin/iconv
drwxr-xr-x  2 root  wheel  opaque     512 Apr  9 10:34 foo/include
-r--r--r--  1 root  wheel  -         4760 Apr  9 09:47 foo/include/iconv.h
-r--r--r--  1 root  wheel  -         1546 Apr  9 09:47 foo/include/libcharset.h
-r--r--r--  1 root  wheel  -         1391 Apr  9 09:47 foo/include/localcharset.h
drwxr-xr-x  2 root  wheel  opaque     512 Apr  9 10:34 foo/lib
-rw-r--r--  1 root  wheel  -         4256 Apr  9 09:47 foo/lib/libcharset.a
-r--r--r--  1 root  wheel  -          807 Apr  9 09:47 foo/lib/libcharset.la
lrwxr-xr-x  1 root  wheel  -           15 Apr  9 09:47 foo/lib/libcharset.so -> libcharset.so.1
-r--r--r--  1 root  wheel  -         8464 Apr  9 09:47 foo/lib/libcharset.so.1
-rw-r--r--  1 root  wheel  -       998722 Apr  9 09:47 foo/lib/libiconv.a
-r--r--r--  1 root  wheel  -          793 Apr  9 09:47 foo/lib/libiconv.la
lrwxr-xr-x  1 root  wheel  -           13 Apr  9 09:47 foo/lib/libiconv.so -> libiconv.so.3
-r--r--r--  1 root  wheel  -      1002230 Apr  9 09:47 foo/lib/libiconv.so.3
drwxr-xr-x  2 root  wheel  opaque     512 Apr  9 10:34 foo/libdata
-r--r--r--  1 root  wheel  -          641 Apr  9 09:47 foo/libdata/charset.alias
drwxr-xr-x  4 root  wheel  opaque     512 Apr  9 10:34 foo/man
drwxr-xr-x  2 root  wheel  opaque     512 Apr  9 10:34 foo/man/man1
-r--r--r--  1 root  wheel  -          976 Apr  9 09:47 foo/man/man1/iconv.1.gz
drwxr-xr-x  2 root  wheel  opaque     512 Apr  9 10:34 foo/man/man3
-r--r--r--  1 root  wheel  -         1457 Apr  9 09:47 foo/man/man3/iconv.3.gz
-r--r--r--  1 root  wheel  -          653 Apr  9 09:47 foo/man/man3/iconv_close.3.gz
-r--r--r--  1 root  wheel  -         2103 Apr  9 09:47 foo/man/man3/iconv_open.3.gz
drwxr-xr-x  3 root  wheel  opaque     512 Apr  9 10:34 foo/share
drwxr-xr-x  3 root  wheel  opaque     512 Apr  9 10:34 foo/share/doc
drwxr-xr-x  2 root  wheel  opaque     512 Apr  9 10:34 foo/share/doc/libiconv
-r--r--r--  1 root  wheel  -         3473 Apr  9 09:47 foo/share/doc/libiconv/iconv.1.html
-r--r--r--  1 root  wheel  -         8223 Apr  9 09:47 foo/share/doc/libiconv/iconv.3.html
-r--r--r--  1 root  wheel  -         2384 Apr  9 09:47 foo/share/doc/libiconv/iconv_close.3.html
-r--r--r--  1 root  wheel  -        16401 Apr  9 09:47 foo/share/doc/libiconv/iconv_open.3.html

The way I see it, bsdtar(1) extracts the symlink libcharset.so, and then
tries to stat(2) instead of lstat(2) it, before libcharset.so.1 is
extracted. The questions is: why?


Ulrich Spoerlein
-- 
"The trouble with the dictionary is you have to know how the word is
spelled before you can look it up to see how it is spelled."
-- Will Cuppy