ASE Home Page Products Download Purchase Support About ASE
ChartDirector Support
 Forum HomeForum Home   SearchSearch

Message ListMessage List     Post MessagePost Message

  missing symbol
Posted by Mikael on Jun-20-2016 22:04
| Reply | Monitor
With the python/linux distribution of ChartDirector, I can't load libchartdir.so because of a missing symbol (with LD_DEBUG=libs):

>>> import pychartdir
...
error: symbol lookup error: undefined symbol: _Unwind_GetIP (fatal)

I'm on a RHEL6 system with python 2.7.11 (gcc 4.7.3). How can I make this work?

Thanks,
Mikael

  Re: missing symbol
Posted by Peter Kwan on Jun-21-2016 01:26
| Reply | Monitor
Hi Mikael,

We have tested ChartDirector with CentOS 6 (which I assume is the same as RHEL6) and it works normally in our case.

Would you mind to provide the following information? We will try to create your environment to reproduce the problem.

(a) Please inform us of the result of "uname -a" so that we can know the exact Linux version you are using and whether it is 32-bit or 64-bit.

(b) Would you mind to inform me of the gcc version in your Linux (use gcc -v)? In our test system, the gcc version is only gcc 4.4. If you system is using a different gcc version, would you mind to clarify how do you obtain the gcc? (Does it come with the OS as standard, or do you obtain it somewhere else or compile it yourself?)

(c) Would you mind to inform me how do you obtain the Python 2.7.11? (Does it come with the OS as standard, or do you obtain it somewhere else or compile it yourself?)

(d) Are you using "fakeroot" or similar systems that isolate your Python environment from the OS?

(e) Please execute the following commands to see if there is any error:

**** NOTE: assume you are in the directory that contains the ChartDirector subdirectory

cd ChartDirector/pythondemo
cp -r ../lib/* .
python simplebar.py

(f) If the above shows the same _Unwind_GetIP error, please in addition try the followings:

export LD_PRELOAD=/lib64/libgcc_s.so.1
python simplebar.py

Please kindly let me know what is the error message.

Regards
Peter Kwan

  Re: missing symbol
Posted by Peter Kwan on Jun-21-2016 14:45
| Reply | Monitor
Hi Mikael,

For your information, I have just downloaded the latest version of CentOS 6 (which is CentOS 6.8 release in May 2016) and try myself. In my case, after a fresh installation, the Python is listed as 2.6.6 with gcc 4.4.7. I tried step (e) above, and there is no error. The chart is created normally. I also download Python 2.7.11 source package and compile them on the CentOS 6.8 using its default gcc 4.4.7, and ChartDirector also works normally with the compiled Python 2.7.11.

Regards
Peter Kwan

  Re: missing symbol
Posted by Mikael on Jun-21-2016 23:59
| Reply | Monitor
Thank you for the speedy reply. It happens on 64bit but not 32bit (I was mistaken about 32bit in my previous message). I notice the following difference:

(64bit)
-bash-4.1$ nm -D libchartdir.so | grep GetIP
                 U _Unwind_GetIP
-bash-4.1$

(32bit)
-bash-4.1$ nm -D libchartdir.so | grep GetIP
-bash-4.1$

Would it be possible to build the 64bit library without this symbol dependency? It's been suggest to me that using gcc 4.7.3 (or later) would accomplish this.

(a)
-bash-4.1$ uname -a
Linux <hostnameremoved> 2.6.32-431.20.3.el6.x86_64 #1 SMP Fri Jun 6 18:30:54 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux

(b) We use gcc 4.7.3.

(c) It is built from source using gcc 4.7.3.

(d) no.

(e) I guess this is as good (if not I will fetch the file you mention):

-bash-4.1$ python -c "import pychartdir"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/******/user64/ext2/g473_linux64/lib/python2.7/site-packages/pychartdir/__init__.py", line 1, in <module>
    from pychartdir import *
  File "/home/******/user64/ext2/g473_linux64/lib/python2.7/site-packages/pychartdir/pychartdir.py", line 5, in <module>
    cdDll = __import__("pychartdir" + cdSysVer[0] + cdSysVer[2])
ImportError: Error loading "/home/******/user64/ext2/g473_linux64/lib/python2.7/site-packages/libchartdir.so" : /home/******/user64/ext2/g473_linux64/lib/python2.7/site-packages/libchartdir.so: cannot open shared object file: No such file or directory

(f)
-bash-4.1$ export LD_PRELOAD=/lib64/libgcc_s.so.1
-bash-4.1$ python -c "import pychartdir"
-bash-4.1$
(ie. OK, no error)

Mikael

  Re: missing symbol
Posted by Peter Kwan on Jun-23-2016 01:38
| Reply | Monitor
Hi Mikael,

ChartDirector has been tested with many versions of gcc. For gcc 4.x, we have tested up to gcc 4.9. For gcc 5.x, we have tested up to gcc 5.3.1.

Although different gcc versions may use different symbols, the Linux OS will not remove old symbols, otherwise a Linux OS will not be compatible with code compiled with older versions of the OS. It may even become incompatible with code compiled with itself after an update is applied (as the update may changed the shared object).

The _Unwind_GetIP the gcc code for handling exceptions in C++ code. It would only cause issues if there is some other errors causing the exception. (You probably already know that Linux uses lazy bindings, which means unused symbols will not cause issues.
I have used LD_DEBUG to check and in my case, the dynamic linker does not bind the _Unwind_GetIP at all.)

For your case, from the directory structure "/home/******/user64/ext2/g473_linux64/lib/python2.7/site-packages/libchartdir.so", I speculate your environment may be is not using the system libraries provided by the OS. Is it possible in your case it is using libraries located at "/home/******/user64/ext2/g473_linux64/lib/"?

You probably know that Linux is itself compiled with gcc, so it has all the gcc libraries and relies on them. If the new gcc is not specifically built to be backwards compatible (such as preserving old symbols), its gcc libraries may conflict with the OS. In this case, the new gcc libraries may need to be kept in a separate directory.

I think compiling ChartDirector specifically for that the gcc is not a good option, because if the gcc is updated again, it will break again. (Every new gcc version can change  some symbols.) The better option is to stick to the OS version of the libraries, which is known to maintain compatibility. In this way, if you change to another machine or update the gcc, the code will still run.

The LD_PRELOAD method is use to force the OS version of the libgcc_s.so to be used, and it does seem to solve the problem.

For your case, if your system is in fact using some sytem libraries in "/home/******/user64/ext2/g473_linux64/lib/", it possible to configure your system to use the libraries provided by the OS?

Regards
Peter Kwan

  Re: missing symbol
Posted by Mikael on Jun-24-2016 21:11
| Reply | Monitor
Again thank you for the reply. We do not believe our system libraries are loaded incorrectly. Can I ask you to try one more thing to reproduce what we are seeing?

In your download, libchartdir.so and pychartdir27.so come in the same directory. Because we link chart director into some of our own c++ built modules, we have (for years) put libchartdir.so in /g473_linux64/lib/ and pychartdir27.so in /g473_linux64/lib/python2.7/site-packages.

If I copy libchartdir.so to site-packages it will, in fact, load without problems. Can you put your (64bit) libchartdir.so on an LD_LIBRARY_PATH away from site-packages?

It looks like the missing symbol is encountered as a consequence (exception handling) of not finding libchartdir.so in site-packages, but only so for the 64bit build.

Mikael

-bash-4.1$ LD_DEBUG=files python -c "import pychartdir27"
(...)
     29645:     calling init: /home/*/user64/ext2/g473_linux64/lib/python2.7/site-packages/pychartdir27.so
     29645:
     29645:     opening file=/home/*/user64/ext2/g473_linux64/lib/python2.7/site-packages/pychartdir27.so [0]; direct_opencount=1
     29645:
     29645:
     29645:     file=/home/*/user64/ext2/g473_linux64/lib/python2.7/site-packages/libchartdir.so [0];  needed by /home/*/user64/ext2/g473_linux64/lib/python2.7/site-packages/pychartdir27.so [0]
     29645:
     29645:     file=libchartdir.so [0];  needed by /home/*/user64/ext2/g473_linux64/lib/python2.7/site-packages/pychartdir27.so [0]
     29645:     file=libchartdir.so [0];  generating link map
     29645:       dynamic: 0x00007f6fe3d30cc0  base: 0x00007f6fe37a2000   size: 0x00000000005a26f8
     29645:         entry: 0x00007f6fe3868b40  phdr: 0x00007f6fe37a2040  phnum:                  5
     29645:
     29645:     /home/*/user64/ext2/g473_linux64/lib/libchartdir.so: error: symbol lookup error: undefined symbol: _Unwind_GetIP (fatal)
     29645:
     29645:     file=/home/*/user64/ext2/g473_linux64/lib/libchartdir.so [0];  destroying link map
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: Error loading "/home/*/user64/ext2/g473_linux64/lib/python2.7/site-packages/libchartdir.so" : /home/*/user64/ext2/g473_linux64/lib/python2.7/site-packages/libchartdir.so: cannot open shared object file: No such file or directory

  Re: missing symbol
Posted by Peter Kwan on Jun-25-2016 03:03
| Reply | Monitor
Hi Mikael,

Thanks for your information. We can now reproduce the problem.

ChartDirector for Python is designed to load "libchartdir.so" from the same directory as "pychartdir??.so", that why in the documentation, it is written that the *everything* in the ChartDirector/lib subdirectory must move together, as they are dependent.

If the "libchartdir.so" is not found in the directory that ChartDirector expects, the OS may try to load it in from OS directories, but the loading may not be successful for various reasons. If the loading is not successful, ChartDirector will report "File Not Found" and specify the directory where ChartDirector expects the "libchartdir.so" to be in. For your case, the "LD_DEBUG" flag reviews the underlying cause to be due to a symbol not found when loading from the OS pathway.

The "libchartdir.so" is shared for all editions of ChartDirector (C++, Python, PHP, .....). Some of the symbols in "libchartdir.so" may only be applicable to C++ but not for Python. The "pychartdir??.so" is specially designed to load "libchartdir.so" so that these unused symbols will not cause issues in Python. However, if the "libchartdir.so" is not in the location that "pychartdir??.so" expects, the OS pathway is used, and it dies when trying to resolve the C++ "_Unwind_GetIP" symbol. Preloading "/lib64/libgcc_s.so" works as this file should be linked in all C++ programs.

One method to solve the problem is to put "libchartdir.so" in the directory that ChartDirector for Python expects. If you need "libchartdir.so" in another directory (eg. because it is used in some C++ programs), you can set one of the "libchartdir.so" to be a symbolic link to the real "libchartdir.so". (The one for Python can be the real file or the symbol link. It does not matter as long as ChartDirector can find it in the expected place.)

We have also created a modified "pychartdir27.so" that can load "libchartdir.so" from OS directories so that unused symbols would not cause issues.

http://www.advsofteng.com/pychartdir27_linux64_602p1.zip

If you use the above modified pychartdir27.so, please let us know if it works in your case, We will do more testing in the meantime, and if it works reliably, we may incorporate it in the next version of ChartDirector.

Regards
Peter Kwan

  Re: missing symbol
Posted by Mikael on Jun-27-2016 21:41
| Reply | Monitor
Thank you, that works like a charm for us. We'd appreciate it being added in future releases.

Mikael