ipython / numpy / scipy weirdness
John W. O'Brien
john at saltant.com
Tue Jun 23 02:54:18 UTC 2015
On 5/27/15 7:14 AM, John W. O'Brien wrote:
> [adding freebsd-python@ back]
>
> On 5/27/15 1:41 AM, Johannes Jost Meixner wrote:
>> On 05/26/2015 16:00, John W. O'Brien wrote:
>>> On 5/22/15 9:57 PM, Johannes Jost Meixner wrote:
>>>> Trying my hand at Data SCIENCE (tm) with a combination of
>>>> ipython notebooks and numpy/scipy, I seem to have stumbled upon
>>>> weirdness we already considered fixed:
>>>>
>>>> ImportError: /lib/libgcc_s.so.1: version GCC_4.6.0 required by
>>>> /usr/local/lib/gcc48/libgfortran.so.3 not found
>>>>
>>>> Supposedly, we have a commit to show for fixing it in
>>>> https://svnweb.freebsd.org/ports?view=revision&revision=369447,
>>>>
>>>> and yet... I can't get numpy nor scipy to work *without* the
>>>> following:
>>>>
>>> [patch]
>>>> % ipython notebook (new site opens in a browser)
>>>>
>>>> create a notebook write into it, 'import numpy' write into it,
>>>> 'import scipy
>>>>
>>>> run the notebook (hit the 'play' button)
>>
>>> I am afraid to say that I fail to reproduce this problem.
>>
>>> % ipython --no-banner
>>
>> I can't produce it with the console. Only with `ipython notebook` and
>> entering those then within a browser. Could you try that?
>
> Yes. In fact, I did it that way first, and then on the console. For what
> it's worth, I only have headless boxen at the ready, so the other run
> looked like this:
>
> fbsd% ipython notebook --no-browser --port=8080
>
> osx% ssh -L 8080:localhost:8080 fbsd
>
> [open http://localhost:8080/ in browser on osx]
>
> It's curious that you see different results between the console and
> browser front-ends. I don't know enough about the ipython architecture
> to infer useful next steps, but that seems significant.
While doing QA on math/py-pandas 0.16.2, I began to investigate a new
test failure in the io.pytables module and promptly stumbled upon the wily
ImportError: /lib/libgcc_s.so.1: version GCC_4.6.0 required by
/usr/local/lib/gcc48/libgfortran.so.3 not found
error. This does not appear to have anything to do with the test failure
I have yet to track down, but since it's relevant to your inquiry from
last month, I wanted to share a new insight I've gained.
On my system, this works fine.
% python
Python 2.7.9 (default, May 13 2015, 01:30:13)
[GCC 4.2.1 Compatible FreeBSD Clang 3.4.1
(tags/RELEASE_34/dot1-final 208032)] on freebsd10
Type "help", "copyright", "credits" or "license" for more information.
>>> from numpy.linalg import _umath_linal, lapack_lite
>>>
But this barfs.
% python
Python 2.7.9 (default, May 13 2015, 01:30:13)
[GCC 4.2.1 Compatible FreeBSD Clang 3.4.1
(tags/RELEASE_34/dot1-final 208032)] on freebsd10
Type "help", "copyright", "credits" or "license" for more information.
>>> import tables
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/site-packages/tables/__init__.py",
line 82, in <module>
from tables.utilsextension import (
File "numpy.pxd", line 155, in init tables.utilsextension
(tables/utilsextension.c:15443)
File "/usr/local/lib/python2.7/site-packages/numpy/__init__.py",
line 170, in <module>
from . import add_newdocs
File
"/usr/local/lib/python2.7/site-packages/numpy/add_newdocs.py", line 13,
in <module>
from numpy.lib import add_newdoc
File
"/usr/local/lib/python2.7/site-packages/numpy/lib/__init__.py", line 18,
in <module>
from .polynomial import *
File
"/usr/local/lib/python2.7/site-packages/numpy/lib/polynomial.py", line
19, in <module>
from numpy.linalg import eigvals, lstsq, inv
File
"/usr/local/lib/python2.7/site-packages/numpy/linalg/__init__.py", line
51, in <module>
from .linalg import *
File
"/usr/local/lib/python2.7/site-packages/numpy/linalg/linalg.py", line
29, in <module>
from numpy.linalg import lapack_lite, _umath_linalg
ImportError: /lib/libgcc_s.so.1: version GCC_4.6.0 required by
/usr/local/lib/gcc48/libgfortran.so.3 not found
>>>
But this works fine.
% python
Python 2.7.9 (default, May 13 2015, 01:30:13)
[GCC 4.2.1 Compatible FreeBSD Clang 3.4.1
(tags/RELEASE_34/dot1-final 208032)] on freebsd10
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> import tables
>>>
Here's what the major players look like behind the scenes.
$ readelf -d
/usr/local/lib/python2.7/site-packages/{numpy/linalg/{_umath_linalg,lapack_lite},tables/utilsextension}.so
\
| egrep "^File|libgcc|libgfortran|rpath"
File:
/usr/local/lib/python2.7/site-packages/numpy/linalg/_umath_linalg.so
0x0000000000000001 (NEEDED) Shared library:
[libgfortran.so.3]
0x0000000000000001 (NEEDED) Shared library: [libgcc_s.so.1]
0x000000000000000f (RPATH) Library rpath:
[/usr/local/lib/gcc48]
File: /usr/local/lib/python2.7/site-packages/numpy/linalg/lapack_lite.so
0x0000000000000001 (NEEDED) Shared library:
[libgfortran.so.3]
0x0000000000000001 (NEEDED) Shared library: [libgcc_s.so.1]
0x000000000000000f (RPATH) Library rpath:
[/usr/local/lib/gcc48]
File: /usr/local/lib/python2.7/site-packages/tables/utilsextension.so
0x0000000000000001 (NEEDED) Shared library: [libgcc_s.so.1]
And here are some examples of the innards from the libraries in question.
% objdump -T /usr/local/lib/gcc48/libgfortran.so.3.0.0 \
| grep GCC_4.6.0 \
| head -3
0000000000000000 DF *UND* 0000000000000000 GCC_4.6.0 __divtf3
0000000000000000 DF *UND* 0000000000000000 GCC_4.6.0 __gttf2
0000000000000000 DF *UND* 0000000000000000 GCC_4.6.0 __eqtf2
% objdump -T /usr/local/lib/gcc48/libgcc_s.so.1 \
| egrep "__(divtf3|gttf2|eqtf2)"
00000000000084e0 g DF .text 0000000000000126 GCC_4.6.0 __eqtf2
0000000000008610 g DF .text 00000000000001b5 GCC_4.6.0 __gttf2
0000000000007a60 g DF .text 0000000000000a7a GCC_4.6.0 __divtf3
% objdump -T /lib/libgcc_s.so.1 \
| egrep "__(divtf3|gttf2|eqtf2)"
What I think I understand now about what this means and how these things
are related is as follows.
When I import tables first, there comes a point when the runtime linker
loads utilsextension.so, sees that it needs libgcc_s.so.1 but doesn't
specify where to find it, loads it from the default /lib, and carries
on... Right up to the point when utilsextensions.so tries to load numpy,
and the runtime linker is presented with the first of _umath_linalg.so
and lapack_lite.so. If I were a runtime linker, I would look at the list
of dependencies, encounter libgcc_s.so.1, see that I've already provided
that to this process, get to the libgfortran.so.3 dependency, and barf
on the missing/inadequate symbols.
When I import numpy first, the runtime linker obeys the rpath and loads
libgcc_s.so.1 from the gcc48 port location, which thereby satisfies
libgfortran.so.3 from the same. At that point, importing tables works
because utilsextensions.so isn't picky about which libgcc_s.so.1 is
offered, so it doesn't matter that the runtime linker hands it the one
from gcc48 instead of the one from base.
The insight this gives me about the general problem is that *any* python
package containing an .so with a requirement on libgcc_s.so.1 that can
be satisfied from base is probably sufficient to make numpy and company
fail with this pesky import error if it happens to be imported first.
I've looked on my system for examples of these with
% find /usr/local/lib/python2.7/site-packages -name "*.so" \
| xargs readelf -d \
| egrep "^File|libgcc|libgfortran|rpath" \
| less
and see a handful that might qualify in scipy, numexpr, matplotlib, and
some others. I spot checked, and none seem to trigger this problem.
Given their common flavor, I wonder whether they all happen to import
numpy internally before loading the potentially-problematic .so module.
However, this could (and does [0]) affect a port that depends indirectly
on numpy (as x11-toolkits/py-wxPython28 does via graphics/py-opengl).
I hope this helps you, and perhaps others who stumble over this. It's
doesn't directly address the scenario you described in your original
message, but perhaps it will give you a new way to think about it.
[0] [PATCH] x11-toolkits/py-wxPython28: Fail embedding_in_wx5.py -
gcc48/libgfortran.so.3 not found
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=196862
Regards,
John
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 827 bytes
Desc: OpenPGP digital signature
URL: <http://lists.freebsd.org/pipermail/freebsd-python/attachments/20150622/ea318dfc/attachment.sig>
More information about the freebsd-python
mailing list