rtld enhancement to add osversion, version 2

[Doug Ambrisko]

This is round 2 of my rtld enhancements.  The primary goal it to have
rtld look in different places for libraries depending on the legacy
binary that we want to run.  This is especially a problem with binaries 
linked to libraries from ports since the version of a library in 
/usr/local/lib is the same whether it was FreeBSD 6, 7, 8 etc. until the 
library itself changes version due to an interface change.  At work we 
need to run 3rd party binaries on our box of which we don't have source.
Having a FreeBSD 6 binaries load /usr/local/lib/libiconv.so.3 that was 
built for FreeBSD 9 is not good.  It is worse when libiconv.so.3 is
linked to libc.so.7 when the FreeBSD 6 binary needs libc.so.6.

I solved this by having rtld extract the OSVERSION from the binary
and then use that to determine what to do.  In my prior version,
I inserted that into the library directory.  That meant to pull
in libc it would look at:
	/lib/603000/libc.so.6
	/lib/6/libc.so.6
	/lib/libc.so.6
to find libc.so.6.  This meant a lot more look ups.  Also I found it
had a problem in that if we had an ambiguous name say
/usr/local/lib/libcrypto.so.5 on the FreeBSD 6 machine and /lib/libcrypto.so.5
on the new FreeBSD 8 system, just doing the search logic would get
a hit on /lib/libcrypto.so.5 before it got to /usr/local/lib/libcrypto.so.5.
So then it would mean we'd have to put all FreeBSD lib's in /lib/6 to
beat the search path.  This wasn't a good solution.  To solve the
performance and path issue, I now follow the 32 bit hints file name 
model.  Now it does a lookup of the hints file based on the osversion
and major.  So now it looks for the hints file as:
	/var/run/ld-elf-603000.so.hints
	/var/run/ld-elf-6.so.hints
	/var/run/ld-elf.so.hints
This is faster and has more unique paths for FreeBSD 6 libraries since
the FreeBSD 6 search paths would be in the hints file.  I modified
ldconfig to accept an "-os=<version>" option to create this hints file.
I tweaked /etc/rc* to make this easy to setup like this:
	ldconfig_os_versions="6"
	ldconfig_6_path="/usr/local/lib/compat/6"

This doesn't solve the LD_LIBRARY_PATH or LD_PRELOAD.  Solving that
I still insert and iterate over the osverion, major and none into the
path to find the library.  The reason for this is that a FreeBSD 8 
binary could exec a FreeBSD 6 binary that execs a FreeBSD 7 binary.  
If for the FreeBSD 6 binary we needed to set a custom LD_LIBRARY_PATH
and the FreeBSD 7 binary find a library via the FreeBSD 6 search path
then the FreeBSD 7 binary would die.  By adding in the osversion search
path I can put the FreeBSD 6 library into the search path + the directory
6.  Then only FreeBSD 6 binaries can find it.  An example:
	LD_LIBRARY_PATH=/usr/custom_software/lib
	/usr/custom_software/lib/6/libfoo.so.6
then only the FreeBSD 6 binary could load it.  Since the searches
would be for the FreeBSD 6 binary:
	/usr/custom_software/lib/603000/libfoo.so.6
	/usr/custom_software/lib/6/libfoo.so.6
	/usr/custom_software/lib/libfoo.so.6
and for FreeBSD 7 binary:
	/usr/custom_software/lib/702000/libfoo.so.6
	/usr/custom_software/lib/7/libfoo.so.6
	/usr/custom_software/lib/libfoo.so.6
Only the FreeBSD 6 binary would load /usr/custom_software/lib/6/libfoo.so.6.
I do the same search with LD_PRELOAD.

Final, is that prior binaries built on FreeBSD i386 but run on FreeBSD amd64
that set LD_* environemnt variables would fail on FreeBSD amd64 as is
since it didn't set the equivalent LD32_*.  To address this for COMPAT_32BIT
I have rtld look for LD32_* first and then check for the LD_* second.
This way legacy applications work fine.

The patch is at:
	http://people.freebsd.org/~ambrisko/rtld_hints_osversion.patch
and has been tested a fair amount at work.  It can make running a
different version of a FreeBSD binary very easy without having any
cross-linking.

There is a bug in FreeBSD 6.0 that didn't record the osversion in the
binary so then this doesn't work.  Try to avoid those binaries!  We 
have a local hack that says if there is no osversion then assume it
is a 6 binary.

Again, this is to solve running legacy binaries on a new version of
FreeBSD of which we don't have source to recompile with new ld features
of -z origin.

Comments?

Thanks,

Doug A.