[wxPython] Opinions sought abt combining the extension modules

wxPython Users
1

Hi all,

As you all probably know, wxPython is actually not just one but several
Python extension modules. There's the core module, wxc, (which is actually
a couple dozen modules all linked toggether,) and there's the add-ons
glcanvasc, oglc, and stc_c. When wxPython is built these are the platform
specific binaries that are the result. Those of you on non-win32 platforms
may also be familiar with libwxPyHelpers.so that gets built. It contains
the code and data that must be shared across all wxPython extension modules.
On win32 this code and data is included in wxc and dynamically linked to by
the other extensions.

While I was converting the build process to use the Python Distutils I
accidentally discovered a way to do the same thing with the helper code on
Linux, (include it in wxc instead of requiring a separate shared library.)
(If you are interested in what the trick was, see below**) Needless to say
this was a happy discovery for me as libwxPyHelpers.so has been the source
of a number of small problems over it's lifetime and besides I wasn't sure I
could get Distutils to make it in the same way it was done before.

Unfortunately this happy-joy-joy time was short lived. While doing testing
for the 2.2.2 release I found that the trick didn't work with Python 2.0!!!
(I got symbol not found errors.) I tried everything I could think of to try
and make it work. I ensured that Python was importing the extension modules
the same as before, I changed the 2.0 makefile so Python would be compiled
and linked with exactly the same flags as 1.5.2, and I added the flags to
the wxc build that specified that all symbols should be exported, all with
no joy. The only thing I havn't tried is to use the same compiler and
linker, (my 1.5.2 binary came from RedHat's RPM,) but I'm starting to get
the idea that this capability is tricky and I probably shouldn't rely upon
it, and it's probably compounded by issues on other platforms, linkers, etc.

First of all, it seems to me, given my understanding of how dynamic linking
works on unix-like platforms, that there should be a way to reliably make
this work. Anybody know how?

Secondly, I really like the idea of doing away with libwxPyHelpers.so, but
the only other way I can think of doing it is to link everything together
into one extension module. With the Distutils this is fairly easy to do,
including allowing options for turning on or off the various add-on modules.
Because of external dependencies I would probably leave the GLCanvas off by
default, but default OGL and STC to on, at least in the binary
distributions. Using the Distutils it should be much easier for people to
(re)build wxPython to change options if needed, (plus it's the new
*standard* way to do Python extensions.) Is anybody opposed to this?
Anybody have a better idea?

···

--
Robin Dunn
Software Craftsman
robin@AllDunn.com
http://wxPython.org Java give you jitters?
http://wxPROs.com Relax with wxPython!

**Basically the thing I did was include the helpers code in wxc, but didn't
explicitly link with it. When the other extension modules were loaded and
resolved all their unresolved symbols they already found those from helpers
in the process memory space from when they were loaded from wxc. (Yes, this
should have been a no-brainer as that is how an extension module finds all
the symbols of the Python API, you don't have to link explicitly with any
library like you do on windows, they are already present in the python
executable already in memory. But why it stopped working is still a
mystery...)

_______________________________________________
wxPython-users mailing list
wxPython-users@lists.sourceforge.net
http://lists.sourceforge.net/mailman/listinfo/wxpython-users

2

Hi Robin,

Did you ever get this resolved? What was the exact symbol(s) it was
complaining about?
I've been recently wrestling with similar problems, so I might have a hunch,
and then again maybe not.

-Harri

···

----- Original Message -----
From: "Robin Dunn" <robin@alldunn.com>
To: "wxPython-users" <wxpython-users@lists.sourceforge.net>
Sent: Wednesday, October 25, 2000 12:53 AM
Subject: [wxPython] Opinions sought abt combining the extension modules

Hi all,

As you all probably know, wxPython is actually not just one but several
Python extension modules. There's the core module, wxc, (which is

actually

a couple dozen modules all linked toggether,) and there's the add-ons
glcanvasc, oglc, and stc_c. When wxPython is built these are the platform
specific binaries that are the result. Those of you on non-win32

platforms

may also be familiar with libwxPyHelpers.so that gets built. It contains
the code and data that must be shared across all wxPython extension

modules.

On win32 this code and data is included in wxc and dynamically linked to

by

the other extensions.

While I was converting the build process to use the Python Distutils I
accidentally discovered a way to do the same thing with the helper code on
Linux, (include it in wxc instead of requiring a separate shared library.)
(If you are interested in what the trick was, see below**) Needless to

say

this was a happy discovery for me as libwxPyHelpers.so has been the source
of a number of small problems over it's lifetime and besides I wasn't sure

I

could get Distutils to make it in the same way it was done before.

Unfortunately this happy-joy-joy time was short lived. While doing

testing

for the 2.2.2 release I found that the trick didn't work with Python

2.0!!!

(I got symbol not found errors.) I tried everything I could think of to

try

and make it work. I ensured that Python was importing the extension

modules

the same as before, I changed the 2.0 makefile so Python would be compiled
and linked with exactly the same flags as 1.5.2, and I added the flags to
the wxc build that specified that all symbols should be exported, all with
no joy. The only thing I havn't tried is to use the same compiler and
linker, (my 1.5.2 binary came from RedHat's RPM,) but I'm starting to get
the idea that this capability is tricky and I probably shouldn't rely upon
it, and it's probably compounded by issues on other platforms, linkers,

etc.

First of all, it seems to me, given my understanding of how dynamic

linking

works on unix-like platforms, that there should be a way to reliably make
this work. Anybody know how?

Secondly, I really like the idea of doing away with libwxPyHelpers.so, but
the only other way I can think of doing it is to link everything together
into one extension module. With the Distutils this is fairly easy to do,
including allowing options for turning on or off the various add-on

modules.

Because of external dependencies I would probably leave the GLCanvas off

by

default, but default OGL and STC to on, at least in the binary
distributions. Using the Distutils it should be much easier for people to
(re)build wxPython to change options if needed, (plus it's the new
*standard* way to do Python extensions.) Is anybody opposed to this?
Anybody have a better idea?

--
Robin Dunn
Software Craftsman
robin@AllDunn.com
http://wxPython.org Java give you jitters?
http://wxPROs.com Relax with wxPython!

**Basically the thing I did was include the helpers code in wxc, but

didn't

explicitly link with it. When the other extension modules were loaded and
resolved all their unresolved symbols they already found those from

helpers

in the process memory space from when they were loaded from wxc. (Yes,

this

should have been a no-brainer as that is how an extension module finds all
the symbols of the Python API, you don't have to link explicitly with any
library like you do on windows, they are already present in the python
executable already in memory. But why it stopped working is still a
mystery...)

_______________________________________________
wxPython-users mailing list
wxPython-users@lists.sourceforge.net
http://lists.sourceforge.net/mailman/listinfo/wxpython-users

_______________________________________________
wxPython-users mailing list
wxPython-users@lists.sourceforge.net
http://lists.sourceforge.net/mailman/listinfo/wxpython-users

3

Hi Robin,

Did you ever get this resolved?

Yes. When Python imports a dynamic extension module it uses

    handle = dlopen(pathname, RTLD_NOW);

but if I changed it to

    handle = dlopen(pathname, RTLD_NOW|RTLD_GLOBAL);

then it did exactly what I wanted, i.e., a dynamically loaded module would
be able to bind to symbols found in previously loaded extension modules. I
rechecked the sources for 1.5.2 and found that it called dlopen the same as
the first one above just like 2.0, so my guess is that when RedHat made
their Python 1.5.2 RPM that they hacked the code to add the RTLD_GLOBAL flag
for some reason.

I decided to back off from my quest for a few reasons:

    1. I don't think I could convince the BDFL to change it
    2. It was too late for 2.0 anyway
    3. I'd lose a lot of credibinity if I made eveybody
       recompile their Python just to use wxPython
    4. It would probably only work on Linux and Solaris as
       most other platforms do dynamic loading differently.

Instead, wxPython 2.2.2 just links all extension modules together into one
.so

What was the exact symbol(s) it was
complaining about?

They were from my helpers.cpp module, which was previously in the
libwxPyHelpers.so and I was trying to make it work by putting it into
wxcmodule.so.

I've been recently wrestling with similar problems, so I might have a

hunch,

and then again maybe not.

HTH,

···

--
Robin Dunn
Software Craftsman
robin@AllDunn.com
http://wxPython.org Java give you jitters?
http://wxPROs.com Relax with wxPython!

_______________________________________________
wxPython-users mailing list
wxPython-users@lists.sourceforge.net
http://lists.sourceforge.net/mailman/listinfo/wxpython-users