math/py-numpy cannot satisfy libgfortran dependency on libgcc_s/GCC_4.6.0 [WAS: Re: py27-pandas-0.13.1_1 conflict with py27-MySQLdb-1.2.3_4]

Tue May 13 03:06:08 UTC 2014

On 5/8/14 11:33 PM, Irjohn Junus wrote:
> Hi John,
> 
> No worries and yes I'm still having the problem.
> 
> I remember seeing this message:
> ===>>> pkg-message for gcc-4.7.3_1
> To ensure binaries built with this toolchain find appropriate versions
> of the necessary run-time libraries, you may want to link using
> 
>   -Wl,-rpath=/usr/local/lib/gcc47
> 
> after upgrading gcc port a few days ago, which is described in your link
> [0]. Do I issue this command as root? Need advice here, thanks.
> 
> Regards,
> Irjohn
> 
> 
> On Fri, May 9, 2014 at 10:42 AM, John W. O'Brien <john at saltant.com
> <mailto:john at saltant.com>> wrote:
> 
>     On 4/18/14 8:45 AM, Irjohn Junus wrote:
>     > Dear All,
>     >
>     > I'm running 9.1-RELEASE and today just after freebsd-update to p11 and
>     > updating ports to the latest found out that pandas won't import after
>     > MySQLdb is imported first. Strangely, it will import no problem if
>     > MySQLdb is imported after:
> 
>     Hi Irjohn,
> 
>     Sorry for the long delay. Are you still having this problem?
> 
>     >>>> import MySQLdb
>     >>>> import pandas as pd
>     > /lib/libgcc_s.so.1: version GCC_4.6.0 required by
>     > /usr/local/lib/gcc47/libgfortran.so.3 not found
>     [...]
>     > Traceback (most recent call last):
>     [...]
>     >   File
>     "/usr/local/lib/python2.7/site-packages/numpy/linalg/linalg.py",
>     > line 29, in <module>
>     >     from numpy.linalg import lapack_lite, _umath_linalg
>     > ImportError: /lib/libgcc_s.so.1: version GCC_4.6.0 required by
>     > /usr/local/lib/gcc47/libgfortran.so.3 not found
> 
>     It looks like this thread [0], these commits [1] [2], and this PR [3]
>     are related.
> 
>     [0]
>     https://lists.freebsd.org/pipermail/freebsd-toolchain/2014-April/001149.html
> 
>     [1] https://svnweb.freebsd.org/ports?view=revision&revision=351167
> 
>     [2] https://svnweb.freebsd.org/ports?view=revision&revision=351159
> 
>     [3] https://www.freebsd.org/cgi/query-pr.cgi?pr=185902

Irjohn,

This is just a little out of my depth, so I'm going to talk through it
out loud while I investigate. Presumably somebody more knowledgeable
than I will step in to correct me if I get something wrong.

First off, there are a few open PRs ([0], [1]) on math/py-numpy, both of
which seem to have been triggered when ports started using GCC 4.7 by
default [2].

What I know from reading gcc(1) is that "-Wl,<option>" passes <option>
to the linker, and from ld(1) is that "-rpath=<path>" embeds <path> in
the output file so that the runtime linker knows where to look for
shared objects. It makes sense why this would help resolve the traceback
you reported because one thing (linalg.py + ??? + libgfortran.so.3) is
looking for another thing (libgcc_s.so.1), finds it, yet remains
unsatisfied. Setting the rpath in the right place in the former will
help it find the right one of the latter.

I dug a little more to learn how to find out what rpath is embedded in a
given file. It turns out that readelf(1) with the "-d" option to show an
ELF file's dynamic section is just the ticket. Here is some output from
my system.

    $ cd /usr/local/lib/gcc47
    $ readelf -d libgfortran.so.3 | grep RPATH
     0x000000000000000f (RPATH)              Library rpath:
[/usr/local/lib/gcc47]
    $ cd /usr/local/lib/python2.7/site-packages/numpy/linalg
    $ readelf -d {_umath_linalg.so,lapack_lite.so} | grep RPATH
     0x000000000000000f (RPATH)              Library rpath:
[/usr/local/lib/gcc47]
     0x000000000000000f (RPATH)              Library rpath:
[/usr/local/lib/gcc47]

In fact, this tool will also tell you which shared objects a file needs.

    $ readelf -d {_umath_linalg.so,lapack_lite.so} | grep libgfortran
     0x0000000000000001 (NEEDED)             Shared library:
[libgfortran.so.3]
     0x0000000000000001 (NEEDED)             Shared library:
[libgfortran.so.3]

So this line of python

    >>> from numpy.linalg import lapack_lite, _umath_linalg

tries to load two binary shared objects, both of which require libgfortran.

Moving right along. If everything goes as it should, how is numpy
supposed to learn the correct rpath? Time to dive into
math/py-numpy/Makefile and the ports machinery.

In math/py-numpy, I find "USES=fortran" and some fishy business in the
pre-configure target involving GCCLIBDIR. The former is pretty
interesting because it pulls in /usr/ports/Mk/Uses/fortran.mk to set all
kinds of useful things, including

    FFLAGS+=        -Wl,-rpath=${LOCALBASE}/lib/gcc${_GCC_VER}
    FCFLAGS+=       -Wl,-rpath=${LOCALBASE}/lib/gcc${_GCC_VER}
    LDFLAGS+=       -Wl,-rpath=${LOCALBASE}/lib/gcc${_GCC_VER} \
                    -L${LOCALBASE}/lib/gcc${_GCC_VER} -B${LOCALBASE}/bin

It also sets FC=gfortran${_GCC_VER}; a fact I can use to further probe
the GCCLIBDIR funny business in the numpy port. During the pre-configure
target, ports determines GCCLIBDIR like this:

    $ gfortran47 -print-file-name=libgfortran.so \
        | sed -e s/libgfortran.so//
    /usr/local/lib/gcc47/gcc/x86_64-portbld-freebsd9.2/4.7.3/../../../

Which simplifies, of course, to /usr/local/lib/gcc47. So far, so good.

I've built math/py-numpy on my machine, and will begin to examine the
build logs [3] for clues about what to dig into next (e.g. differences
between my setting CC=clang et al, and not). In the mean time...

Questions to you:

1.  What is the rpath stored in your copy of those numpy files? That
    is, what is the output from

        cd /usr/local/lib/python2.7/site-packages/numpy/linalg
        readelf -d _umath_linalg.so | grep RPATH
        readelf -d lapack_lite.so | grep RPATH

    ?

2.  What is the contents of
/usr/local/lib/python2.7/site-packages/numpy/distutils/site.cfg on your
machine?

2.  What is the contents of /etc/make.conf on your machine?

3.  How to you install and upgrade your ports? Build from source by
    hand? Build from source with the help of portmaster, poudriere, etc?
    Install binary packages with pkg_* or ports-mgmt/pkg (pkgng)?

One thing that still puzzles me is what databases/py-MySQLdb would have
to do with it.

[0] ports/188114: math/py-numpy is broken on 9.2-STABLE
    http://www.freebsd.org/cgi/query-pr.cgi?pr=ports/188114
[1] ports/188327: math/py-numpy: Numpy is broken
    http://www.freebsd.org/cgi/query-pr.cgi?pr=ports/188327
[2] https://svnweb.freebsd.org/ports?view=revision&revision=347809
[3]
https://pkg.saltant.net/poudriere/bulk/92amd64-current/2014-05-12_21h56m01s/logs/py27-numpy-1.8.0_1,1.log

Regards,
John

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: OpenPGP digital signature
URL: <http://lists.freebsd.org/pipermail/freebsd-python/attachments/20140512/1b4c61a5/attachment.sig>