rtld enhancement to add osversion, version 2

Mon Mar 19 21:03:59 UTC 2012

On Mon, Mar 19, 2012 at 04:59:34PM -0400, John Baldwin wrote:
> On Tuesday, March 13, 2012 1:29:17 pm Doug Ambrisko wrote:
> > This is round 2 of my rtld enhancements.  The primary goal it to have
> > rtld look in different places for libraries depending on the legacy
> > binary that we want to run.  This is especially a problem with binaries 
> > linked to libraries from ports since the version of a library in 
> > /usr/local/lib is the same whether it was FreeBSD 6, 7, 8 etc. until the 
> > library itself changes version due to an interface change.  At work we 
> > need to run 3rd party binaries on our box of which we don't have source.
> > Having a FreeBSD 6 binaries load /usr/local/lib/libiconv.so.3 that was 
> > built for FreeBSD 9 is not good.  It is worse when libiconv.so.3 is
> > linked to libc.so.7 when the FreeBSD 6 binary needs libc.so.6.
> > 
> > I solved this by having rtld extract the OSVERSION from the binary
> > and then use that to determine what to do.  In my prior version,
> > I inserted that into the library directory.  That meant to pull
> > in libc it would look at:
> > 	/lib/603000/libc.so.6
> > 	/lib/6/libc.so.6
> > 	/lib/libc.so.6
> > to find libc.so.6.  This meant a lot more look ups.  Also I found it
> > had a problem in that if we had an ambiguous name say
> > /usr/local/lib/libcrypto.so.5 on the FreeBSD 6 machine and 
> /lib/libcrypto.so.5
> > on the new FreeBSD 8 system, just doing the search logic would get
> > a hit on /lib/libcrypto.so.5 before it got to /usr/local/lib/libcrypto.so.5.
> > So then it would mean we'd have to put all FreeBSD lib's in /lib/6 to
> > beat the search path.  This wasn't a good solution.  To solve the
> > performance and path issue, I now follow the 32 bit hints file name 
> > model.  Now it does a lookup of the hints file based on the osversion
> > and major.  So now it looks for the hints file as:
> > 	/var/run/ld-elf-603000.so.hints
> > 	/var/run/ld-elf-6.so.hints
> > 	/var/run/ld-elf.so.hints
> > This is faster and has more unique paths for FreeBSD 6 libraries since
> > the FreeBSD 6 search paths would be in the hints file.  I modified
> > ldconfig to accept an "-os=<version>" option to create this hints file.
> > I tweaked /etc/rc* to make this easy to setup like this:
> > 	ldconfig_os_versions="6"
> > 	ldconfig_6_path="/usr/local/lib/compat/6"
> 
> I think this is a definite improvement from before, thanks!
> 
> > This doesn't solve the LD_LIBRARY_PATH or LD_PRELOAD.  Solving that
> > I still insert and iterate over the osverion, major and none into the
> > path to find the library.  The reason for this is that a FreeBSD 8 
> > binary could exec a FreeBSD 6 binary that execs a FreeBSD 7 binary.  
> > If for the FreeBSD 6 binary we needed to set a custom LD_LIBRARY_PATH
> > and the FreeBSD 7 binary find a library via the FreeBSD 6 search path
> > then the FreeBSD 7 binary would die.  By adding in the osversion search
> > path I can put the FreeBSD 6 library into the search path + the directory
> > 6.  Then only FreeBSD 6 binaries can find it.  An example:
> > 	LD_LIBRARY_PATH=/usr/custom_software/lib
> > 	/usr/custom_software/lib/6/libfoo.so.6
> > then only the FreeBSD 6 binary could load it.  Since the searches
> > would be for the FreeBSD 6 binary:
> > 	/usr/custom_software/lib/603000/libfoo.so.6
> > 	/usr/custom_software/lib/6/libfoo.so.6
> > 	/usr/custom_software/lib/libfoo.so.6
> > and for FreeBSD 7 binary:
> > 	/usr/custom_software/lib/702000/libfoo.so.6
> > 	/usr/custom_software/lib/7/libfoo.so.6
> > 	/usr/custom_software/lib/libfoo.so.6
> > Only the FreeBSD 6 binary would load /usr/custom_software/lib/6/libfoo.so.6.
> > I do the same search with LD_PRELOAD.
> 
> Hmm, I'm still not quite fan of this.  Perhaps you could add an extension to
> ldconfig and the hints file to handle this case?  That is, a way to store
> path mappings so you could do something like:
> 
> ldconfig -os=6 -p /usr/local/lib /usr/local/lib/6
> 
> Or maybe you could make it an extension of how 'm' worked, so you could make
> directories accept an optional set of colon-separated paths that they serve
> as aliases for:
> 
> ldconfig -os=6 -m /usr/local/lib/6:/usr/local/lib:/usr/lib
> 
> (That would even fit into your existing rc.d script changes I believe).  Then
> rtld would keep this internal directory mapping and be able to map the
> '/usr/local/lib' and '/usr/lib' directories in LD_PRELOAD and LD_LIBRARY_PATH
> to /usr/local/lib/6.  The advantage of this is the same as with your previous
> change that all the mappings are configurable and not hard-coded into rtld
> itself.
> 
> > Final, is that prior binaries built on FreeBSD i386 but run on FreeBSD amd64
> > that set LD_* environemnt variables would fail on FreeBSD amd64 as is
> > since it didn't set the equivalent LD32_*.  To address this for COMPAT_32BIT
> > I have rtld look for LD32_* first and then check for the LD_* second.
> > This way legacy applications work fine.
> 
> Hmm, so Yahoo! had some patches to handle this as well.  I think Yahoo's
> patches were different though.  They actually patched the 32-bit libc to
> capture attempts to get or set LD_* and convert them to actually get/set
> LD32_* instead.  I'm not sure which approach is best, but it might be worth
> asking Peter why Yahoo! did it that way and if there were reasons for that
> approach vs. doing it in rtld.  I think the primary reason was so that you
> could set LD_LIBRARY_PATH or LD_PRELOAD to reference 64-bit libraries and
> not have it break 32-bit apps, but if a 32-bit app tried to set a variable
> before launching another app it would still DTRT.
> 
> I do think this is definitely a problem worth solving.

Just a trivial note: r232831 added note parsing and stores osrel in
the Obj_Entry.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 196 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-arch/attachments/20120319/4e48a4a3/attachment.pgp