wxPython extensions and swig directors together?

Hi, I have been doing some wxWidgets work in C++ and driving it
with top level wxPython. Now I need my python code to react to
events in the C++ world. I'm trying to use SWIG's directors, and
they work, unless some other part of my extension does an %import core.i

Once I import the core.i of wxPython, my directors crash hard. (Exception
reading from 0x0000008)

Has anyone had any luck using swig directors and wxPython together?

If I can't use directors, how can I roll my own in a wxPython-safe way?
I need to override a C++ method in Python so that when C++ 'calls back'
into my object, my python code runs.

details here:
http://sourceforge.net/mailarchive/forum.php?thread_id=25616412&forum_id=46074

I'm going to try to put together a minimal extension that demonstrates the crash.

-Jim

Has anyone had any luck using swig directors and
wxPython together?

I'm going to try to put together a minimal extension that
demonstrates the crash.

Here's an extension that combines the swig 'callback' example
with the wxPython Wiki foobutton extension and uses distutils
to do the build.

  http://www.maplesong.com/wxPyDirectors.zip

Details are in the readme.txt

Am I doing something wrong to cause the crash?

Thanks,
-Jim

Jim wrote:

Has anyone had any luck using swig directors and wxPython together?

I'm going to try to put together a minimal extension that demonstrates the crash.

Here's an extension that combines the swig 'callback' example with the wxPython Wiki foobutton extension and uses distutils to do the build.

  http://www.maplesong.com/wxPyDirectors.zip

Details are in the readme.txt

Am I doing something wrong to cause the crash?

Thanks for the sample. I'll try to dig into it later today and see what's going on.

···

--
Robin Dunn
Software Craftsman
http://wxPython.org Java give you jitters? Relax with wxPython!

Thanks for the sample. I'll try to dig into it later today and see
what's going on.

Thanks Robin. You rock.

-Jim

Jim wrote:

Hi, I have been doing some wxWidgets work in C++ and driving it
with top level wxPython. Now I need my python code to react to events in the C++ world. I'm trying to use SWIG's directors, and they work, unless some other part of my extension does an %import core.i

The problem appears to be related to the fact that wxPython uses SWIG directives to wrap all calls to C/C++ functions with code that releases and reacquires the Python Global Interpreter Lock, in order to allow Python code in other threads to run while the C/C++ function is running. In _defs.i you;ll find this code:

%exception {
     PyThreadState* __tstate = wxPyBeginAllowThreads();
     $action
     wxPyEndAllowThreads(__tstate);
     if (PyErr_Occurred()) SWIG_fail;
}

When the GIL is released like this then any code inside of $action that manipulates PyObjects, (like building a list if items for a return value) or calls Python code (like an event handler or a virtual method callback) needs to acquire the GIL before it can do so. If not then Python will abort() if it is able or I've seen it just crash a couple times too.

So basically the problem is that when you %import core.i you are getting the above declaration and so all your function calls will release and reacquire the GIL when they run, but your virtual method callback wrappers generated by the swig director feature are not acquiring the GIL before they try to call the Python method in the derived class.

There are a few approaches to take to solve this.

* Rebuild wxPython with threads disabled by undefining WXP_WITH_THREAD. Not recommended as it will prevent other threads from running while any wx C/C++ code is running.

* Turn off the above declaration for your director based classes by using the empty %exception directive in your .i files. Care should be taken with this approach as well because if anything done in your virtual callbacks causes a wx event to be sent or wx callback to be called then the code there will expect that it needs to reacquire the GIL and it could cause a deadlock (although the risk of this is greatly reduced in Python 2.4+ because of the use of a newer Python API for ensuring that the GIL is acquired safely.)

* Add SWIG directives that will add code to the director functions that invoke the Python methods, such that the call to the Python function is wrapped in code like this:

  wxPyBlock_t blocked = wxPyBeginBlockThreads();
  ...
  wxPyEndBlockThreads(blocked);
  
Having not done more than experiment a bit with directors I'm not sure how to do that, but I expect that there is a way.

···

--
Robin Dunn
Software Craftsman
http://wxPython.org Java give you jitters? Relax with wxPython!

Robin Dunn <robin <at> alldunn.com> writes:

So basically the problem is that when you %import core.i you are getting
the above declaration and so all your function calls will release and
reacquire the GIL when they run, but your virtual method callback
wrappers generated by the swig director feature are not acquiring the
GIL before they try to call the Python method in the derived class.

That doesn't sound too bad. I'll try your suggestions, and If I
can use %exception in just the adapters, I'll be very happy.

For now, I've broken my extension into two extensions, and one
does directors and one extends wxPython and I'm back in business.

Soon I will want to mix the two in the same class, and my build
will get much simpler if I can reduce it back to one module as
well...

I'll let you know how it goes.

----- and now for something completely different -------

What did you think of the distutils setup.py for the
extenion building? Is it Wiki material? I was psyched to
get it working.

Thanks,
-Jim

Jim wrote:

Robin Dunn <robin <at> alldunn.com> writes:

----- and now for something completely different -------

What did you think of the distutils setup.py for the extenion building?

It was good. I had to make a couple changes to get it to work for me though.

1. I wanted to use my wxPython project dir for the source instead of the wxPython-win32-devel tarball, so in addition to changing wxpy_devel_dir I needed to add os.path.join(wxpy_devel_dir, "wxPython/include") to includeDirs and add a -I%s/wxPython/src to swig_opts

2. Since I wanted to build for the debug version I needed to replace the 'h' with 'd' in the lib names and in the vc_dll subdir.

3. My swig 1.3.27 is not the one on the path so I needed to add a 'swig' item to the build_ext options to give it the full path name to the swig.exe to use.

Is it Wiki material?

I think so, with a few tweaks to make it more general purpose. For example #2 above could be fixed in the code by checking for --debug or -g on the command line.

Also, you can import wx.build.config and get lots of things directly from wxPython, like the list of includes, defines, libs, swig flags, etc., plus a bunch of functions to help with various parts of the build.

···

--
Robin Dunn
Software Craftsman
http://wxPython.org Java give you jitters? Relax with wxPython!