rtld enhancement to add osversion, version 2

[Doug Ambrisko]

John Baldwin writes:
| On Tuesday, March 13, 2012 1:29:17 pm Doug Ambrisko wrote:
| > This is round 2 of my rtld enhancements.  The primary goal it to have
| > rtld look in different places for libraries depending on the legacy
| > binary that we want to run.  This is especially a problem with binaries 
| > linked to libraries from ports since the version of a library in 
| > /usr/local/lib is the same whether it was FreeBSD 6, 7, 8 etc. until the 
| > library itself changes version due to an interface change.  At work we 
| > need to run 3rd party binaries on our box of which we don't have source.
| > Having a FreeBSD 6 binaries load /usr/local/lib/libiconv.so.3 that was 
| > built for FreeBSD 9 is not good.  It is worse when libiconv.so.3 is
| > linked to libc.so.7 when the FreeBSD 6 binary needs libc.so.6.
| > 
| > I solved this by having rtld extract the OSVERSION from the binary
| > and then use that to determine what to do.  In my prior version,
| > I inserted that into the library directory.  That meant to pull
| > in libc it would look at:
| > 	/lib/603000/libc.so.6
| > 	/lib/6/libc.so.6
| > 	/lib/libc.so.6
| > to find libc.so.6.  This meant a lot more look ups.  Also I found it
| > had a problem in that if we had an ambiguous name say
| > /usr/local/lib/libcrypto.so.5 on the FreeBSD 6 machine and 
| /lib/libcrypto.so.5
| > on the new FreeBSD 8 system, just doing the search logic would get
| > a hit on /lib/libcrypto.so.5 before it got to /usr/local/lib/libcrypto.so.5.
| > So then it would mean we'd have to put all FreeBSD lib's in /lib/6 to
| > beat the search path.  This wasn't a good solution.  To solve the
| > performance and path issue, I now follow the 32 bit hints file name 
| > model.  Now it does a lookup of the hints file based on the osversion
| > and major.  So now it looks for the hints file as:
| > 	/var/run/ld-elf-603000.so.hints
| > 	/var/run/ld-elf-6.so.hints
| > 	/var/run/ld-elf.so.hints
| > This is faster and has more unique paths for FreeBSD 6 libraries since
| > the FreeBSD 6 search paths would be in the hints file.  I modified
| > ldconfig to accept an "-os=<version>" option to create this hints file.
| > I tweaked /etc/rc* to make this easy to setup like this:
| > 	ldconfig_os_versions="6"
| > 	ldconfig_6_path="/usr/local/lib/compat/6"
| 
| I think this is a definite improvement from before, thanks!
| 
| > This doesn't solve the LD_LIBRARY_PATH or LD_PRELOAD.  Solving that
| > I still insert and iterate over the osverion, major and none into the
| > path to find the library.  The reason for this is that a FreeBSD 8 
| > binary could exec a FreeBSD 6 binary that execs a FreeBSD 7 binary.  
| > If for the FreeBSD 6 binary we needed to set a custom LD_LIBRARY_PATH
| > and the FreeBSD 7 binary find a library via the FreeBSD 6 search path
| > then the FreeBSD 7 binary would die.  By adding in the osversion search
| > path I can put the FreeBSD 6 library into the search path + the directory
| > 6.  Then only FreeBSD 6 binaries can find it.  An example:
| > 	LD_LIBRARY_PATH=/usr/custom_software/lib
| > 	/usr/custom_software/lib/6/libfoo.so.6
| > then only the FreeBSD 6 binary could load it.  Since the searches
| > would be for the FreeBSD 6 binary:
| > 	/usr/custom_software/lib/603000/libfoo.so.6
| > 	/usr/custom_software/lib/6/libfoo.so.6
| > 	/usr/custom_software/lib/libfoo.so.6
| > and for FreeBSD 7 binary:
| > 	/usr/custom_software/lib/702000/libfoo.so.6
| > 	/usr/custom_software/lib/7/libfoo.so.6
| > 	/usr/custom_software/lib/libfoo.so.6
| > Only the FreeBSD 6 binary would load /usr/custom_software/lib/6/libfoo.so.6.
| > I do the same search with LD_PRELOAD.
| 
| Hmm, I'm still not quite fan of this.  Perhaps you could add an extension to
| ldconfig and the hints file to handle this case?  That is, a way to store
| path mappings so you could do something like:
| 
| ldconfig -os=6 -p /usr/local/lib /usr/local/lib/6

I'll have to look to see how the hints file could update rtld.  It is
an interesting idea.  Maybe libmap.conf would be better place for this.
I haven't looked at how libmap works.  Maybe introduce:
	/etc/libmap-<OSVERSION>.conf
that maps paths as well.  So with the above example.
	/etc/libmap-6.conf
would contain
	/usr/custom_software/lib /usr/custom_software/lib/6:/usr/custom_software/lib
for example.
 
| Or maybe you could make it an extension of how 'm' worked, so you could make
| directories accept an optional set of colon-separated paths that they serve
| as aliases for:
| 
| ldconfig -os=6 -m /usr/local/lib/6:/usr/local/lib:/usr/lib

I don't really get how this is solving the LD_LIBRARY_PATH/LD_PRELOAD since
"-m" is solving the general case with the hints file which I did first.
 
| (That would even fit into your existing rc.d script changes I believe).  Then
| rtld would keep this internal directory mapping and be able to map the
| '/usr/local/lib' and '/usr/lib' directories in LD_PRELOAD and LD_LIBRARY_PATH
| to /usr/local/lib/6.  The advantage of this is the same as with your previous
| change that all the mappings are configurable and not hard-coded into rtld
| itself.
 
| > Final, is that prior binaries built on FreeBSD i386 but run on FreeBSD amd64
| > that set LD_* environemnt variables would fail on FreeBSD amd64 as is
| > since it didn't set the equivalent LD32_*.  To address this for COMPAT_32BIT
| > I have rtld look for LD32_* first and then check for the LD_* second.
| > This way legacy applications work fine.
| 
| Hmm, so Yahoo! had some patches to handle this as well.  I think Yahoo's
| patches were different though.  They actually patched the 32-bit libc to
| capture attempts to get or set LD_* and convert them to actually get/set
| LD32_* instead.  I'm not sure which approach is best, but it might be worth
| asking Peter why Yahoo! did it that way and if there were reasons for that
| approach vs. doing it in rtld.  I think the primary reason was so that you
| could set LD_LIBRARY_PATH or LD_PRELOAD to reference 64-bit libraries and
| not have it break 32-bit apps, but if a 32-bit app tried to set a variable
| before launching another app it would still DTRT.

This means you would have to have a modified libc in these environment
and it wouldn't help in the static binary case.  I think we are safe
in the case LD_ for a 32 bit binary effecting a 64 bit since rtld should
not allow loading a the wrong type of lib. into a binary.  I can do some
more testing around that area.  I was trying to keep all changes in the
host environment so we can run unchanged binaries from other vendors.
 
| I do think this is definitely a problem worth solving.

Thanks,

Doug A.