My C++ extension hangs when called from within wxPython

Hi again,

I am not running the interpreter interactively but through a script.
In the script I run the extension (which is a long running task) in a
seperate thread using wx.lib.delayedresult. The sample code below
produces the behavior i describe. The first block of code is just to
set up the extension and not related to the problem. I run it on OSX.
The extension is C++ and the interface is generated using SWIG. It is
compiled using the C++ compiler through setuptools. Thanks again.

Kasper

···

########################################################
import Options
optionParser = Options.Options()
optionParser.options.project = '/Users/kasper/Desktop/guitest'
optionParser.postProcess()
from SAP.Sampling import Barcoder as plugin
extension = plugin.Sampler(optionParser.options)
########################################################

import wx

class MyFrame(wx.Frame):
    def __init__(self, *args, **kwds):
        kwds["style"] = wx.DEFAULT_FRAME_STYLE
        wx.Frame.__init__(self, *args, **kwds)

class MyApp(wx.App):

    def OnInit(self):
        wx.InitAllImageHandlers()
        frame_1 = MyFrame(None, -1, "")
        frame_1.CenterOnScreen()
        self.SetTopWindow(frame_1)

        frame_1.Show()
        return 1

def start_gui():

    # If I call run method from the extension here it runs fine:
    extension.run('/Users/kasper/Desktop/guitest/alignmentcache/test_50262771_sediment_one.nex')

    app = MyApp(0)

    # But if i call it here (or anywhere else after this point) the
whole thing hangs,
    # I get "Python (Not Resonding)" in the Activity Monitor, and the
Python.app icon
    # keeps jumping in the dock likek it is trying to start up without
succeeding.
    extension.run('/Users/kasper/Desktop/guitest/alignmentcache/test_50262771_sediment_one.nex')

    app.MainLoop()

if __name__ == "__main__":

   start_gui()

Nathaniel Echols wrote:

    I have written a nice GUI (very easy in wxPython). The GUI is meant to
    run a task implemented as a C++ extension using SWIG. My extension
    works fine when called before I invoke wxPython by subclassing wx.App,
    but if I call it after, as I would obviously like to, it hangs (the
    interpreter becomes unresponsive). Is there some reason why I can't
    call a C++ extension form within wxPython? - And if so what can I do?

It's not clear to me what phenomenon you're describing; are you
running the interpreter interactively, and is this happening after you
run App.MainLoop()? If it's simply a matter of the interpreter not
dying when you hit ctrl-C, that could be the fault of a bare 'except'
clause. I call various custom C++ extensions (using Boost.Python)
from wxPython all the time without a problem, but I often (though not
always) have to use threads to "detach" these from the main GUI
process, and I make sure to let KeyboardInterrupt exceptions through.
(wxPython itself is a C++ extension that uses SWIG [I think], so
there's no technical reason why what you want to do isn't possible.)

Can you send some sample code?

-Nat

wxPython is a wrapper of the C++ wx GUI package. Robin "wraps" it
through the SWIG process. Anyway, it sounds like the OP is running a
process that is blocking wx's mainloop. Check out the wiki here:

http://wiki.wxpython.org/LongRunningTasks

-------------------
Mike Driscoll

--
Department of Integrative Biology
University of California, Berkeley
Office phone: (510) 643-6299

Kasper Munch wrote:
> The sample code below

produces the behavior i describe.

I think the issue here is that you aren't thinking in terms of an event driven app -- when wx.app.MainLoop is called, the vent loop is started, and then EVERYTHING that you want to happen should be as the result of an event.

You might want to try this by having your extension code started in a button handler, for instance.

Another option is wx.CallAfter() which will put an event on the stack to be be run after all the current events are handled. I think you could put it at the end of your Frame __init__ method, for instance, or maybe just before calling MainLoop().

-Chris

···

--
Christopher Barker, Ph.D.
Oceanographer

NOAA/OR&R/HAZMAT (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception

Maybe the sample code got a little to simple.

In the real script a button event executes a onRunButton() function
that then sets up a separate thread using wx.lib.delayedresult where
the extension then called from.

Thanks,
Kasper

···

On Wed, Sep 24, 2008 at 3:16 PM, Christopher Barker <Chris.Barker@noaa.gov> wrote:

Kasper Munch wrote:

The sample code below

produces the behavior i describe.

I think the issue here is that you aren't thinking in terms of an event
driven app -- when wx.app.MainLoop is called, the vent loop is started, and
then EVERYTHING that you want to happen should be as the result of an event.

You might want to try this by having your extension code started in a button
handler, for instance.

Another option is wx.CallAfter() which will put an event on the stack to be
be run after all the current events are handled. I think you could put it at
the end of your Frame __init__ method, for instance, or maybe just before
calling MainLoop().

-Chris

--
Christopher Barker, Ph.D.
Oceanographer

NOAA/OR&R/HAZMAT (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception
_______________________________________________
wxpython-users mailing list
wxpython-users@lists.wxwidgets.org
http://lists.wxwidgets.org/mailman/listinfo/wxpython-users

--
Department of Integrative Biology
University of California, Berkeley
Office phone: (510) 643-6299

Kasper Munch wrote:

Maybe the sample code got a little to simple.

there is that problem with simplifying...

In the real script a button event executes a onRunButton() function
that then sets up a separate thread using wx.lib.delayedresult where
the extension then called from.

Is there any code calling into the python API in that extension? IN particular, anything that would invoke the GIL?

-Chris

···

--
Christopher Barker, Ph.D.
Oceanographer

NOAA/OR&R/HAZMAT (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception

When you say python API, do you mean wxPython? - And what is GIL?

Appreciate your help,
Kasper

···

On Wed, Sep 24, 2008 at 3:40 PM, Christopher Barker <Chris.Barker@noaa.gov> wrote:

Kasper Munch wrote:

Maybe the sample code got a little to simple.

there is that problem with simplifying...

In the real script a button event executes a onRunButton() function
that then sets up a separate thread using wx.lib.delayedresult where
the extension then called from.

Is there any code calling into the python API in that extension? IN
particular, anything that would invoke the GIL?

-Chris

--
Christopher Barker, Ph.D.
Oceanographer

NOAA/OR&R/HAZMAT (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception
_______________________________________________
wxpython-users mailing list
wxpython-users@lists.wxwidgets.org
http://lists.wxwidgets.org/mailman/listinfo/wxpython-users

--
Department of Integrative Biology
University of California, Berkeley
Office phone: (510) 643-6299

Kasper Munch wrote:

When you say python API, do you mean wxPython? - And what is GIL?

sorry -- I mean calling back into the python interpreter at all. GIL is the Global Interpreter Lock -- there are a number of python operations (I can't tell you which!) that require the interpreter to lock so that multiple threads don't et all hung up on each-other -- it can kill multi-threaded performance.

Though if your extension calls wx, there will be other threading issues -- all wx functions must be run in the same thread, except a few exceptions, like wx.PostEvent and wxCallAfter.

Maybe the sample code got a little to simple.

I"d try to make a simple-as=possible example that you can run and demonstrate the problem -- someone is more likely to be help if they can see that.

-Chris

···

--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@noaa.gov

Here is some code that procudes the same problem. The extension is
called from a worker thread which is called from a method triggered by
a button event. This captures in escense what I am doing. I hope i
also explains to someone why the python interpreter then goes into
some kind of death roll. In the Activity Monitor (on my mac) I get
"Python (Not Responding)". The extension runs fine when called form
outside wxpython. I run it on the command line line this "python
simple.py". I noticed that when wx.App is called Python.app (osx
application) takes over from the command line python. They should be
part of the same distribution so I don't see why that should matter.
Any help is appreciated.

import sys, re, os, copy, time, glob
from optparse import OptionParser

# My own class for processing options:
import Options
from SAP.Sampling import Barcoder as plugin

import wx
import wx.lib.scrolledpanel as scrolled
import wx.lib.delayedresult as delayedresult

# Parsing options (required setup code to before calling the extension):
optionParser = Options.Options()
optionParser.options.project = '/Users/kasper/Desktop/guitest'
optionParser.postProcess()

class MyFrame(wx.Frame):

    def __init__(self, *args, **kwds):

        kwds["style"] = wx.DEFAULT_FRAME_STYLE
        wx.Frame.__init__(self, *args, **kwds)

        # Job ids for worker threads:
        self.jobID = 0

        self.panel = panel = scrolled.ScrolledPanel(self, -1, style =
wx.TAB_TRAVERSAL, name="panel1" )
        tabSizer = wx.BoxSizer(wx.VERTICAL)

        self.runB = runB = wx.Button(panel, -1, "Run")
        runB.SetDefault()
        self.Bind(wx.EVT_BUTTON, self.onRunButton, runB)

        tabSizer.Add(runB)
        panel.SetSizer(tabSizer)
        panel.SetAutoLayout(True)
        panel.SetupScrolling()

        frameSizer = wx.BoxSizer(wx.VERTICAL)
        frameSizer.SetMinSize((100, 100))

        frameSizer.Add(panel, 1, wx.EXPAND|wx.ALL, 10)
        self.SetSizer(frameSizer)

        frameSizer.Fit(self)

        self.Layout()

    def onRunButton(self, evt):

        self.jobID += 1
        delayedresult.startWorker(self._resultConsumer, self._resultProducer,
                                  wargs=(self.jobID,
delayedresult.AbortEvent()), jobID=self.jobID)

    def _resultProducer(self, jobID, abortEvent):

       global optionParser

       # This is where I call the extension:
       sampler = plugin.Sampler(optionParser.options)
       sampler.run('/Users/kasper/Desktop/guitest/alignmentcache/test_50262771_sediment_one.nex')

    def _resultConsumer(self, delayedResult):

        jobID = delayedResult.getJobID()
        result = delayedResult.get()

class MyApp(wx.App):

    def OnInit(self):

        wx.InitAllImageHandlers()
        frame_1 = MyFrame(None, -1, "")
        frame_1.CenterOnScreen()
        self.SetTopWindow(frame_1)

        frame_1.Show()
        return 1

def start_gui():
    app = MyApp(0)
    app.MainLoop()

if __name__ == "__main__":

# # If I call the extension here it works fine...
# sampler = plugin.Sampler(optionParser.options)
# sampler.run('/Users/kasper/Desktop/guitest/alignmentcache/test_50262771_sediment_one.nex')

   start_gui()

···

On Wed, Sep 24, 2008 at 11:10 PM, Christopher Barker <Chris.Barker@noaa.gov> wrote:

Kasper Munch wrote:

When you say python API, do you mean wxPython? - And what is GIL?

sorry -- I mean calling back into the python interpreter at all. GIL is the
Global Interpreter Lock -- there are a number of python operations (I can't
tell you which!) that require the interpreter to lock so that multiple
threads don't et all hung up on each-other -- it can kill multi-threaded
performance.

Though if your extension calls wx, there will be other threading issues --
all wx functions must be run in the same thread, except a few exceptions,
like wx.PostEvent and wxCallAfter.

Maybe the sample code got a little to simple.

I"d try to make a simple-as=possible example that you can run and
demonstrate the problem -- someone is more likely to be help if they can see
that.

-Chris

--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@noaa.gov
_______________________________________________
wxpython-users mailing list
wxpython-users@lists.wxwidgets.org
http://lists.wxwidgets.org/mailman/listinfo/wxpython-users

--
Department of Integrative Biology
University of California, Berkeley
Office phone: (510) 643-6299

Christopher Barker wrote:

Kasper Munch wrote:

When you say python API, do you mean wxPython? - And what is GIL?

sorry -- I mean calling back into the python interpreter at all. GIL is the Global Interpreter Lock -- there are a number of python operations (I can't tell you which!) that require the interpreter to lock so that multiple threads don't et all hung up on each-other -- it can kill multi-threaded performance.

Basically it boils down to this: Only one thread can be executing Python byte-code or manipulating Python objects in any other way at one time. So if you make a call into an extension module and it does not release the GIL then *no* other Python code (including most Python C APIs that can result in Python code being executed or Python objects being created) can be executed until that extension call returns.

wxPython releases the GIL on the way in to most[1] of the wx wrappers, reacquires it on the way back out, and also acquires it before calling event handlers and other callbacks. Kasper, if your extension module does not release the GIL then it could behave like what you have been describing.

[1] Except for lightweight things like dealing with wx.Point objects, etc.

···

--
Robin Dunn
Software Craftsman
http://wxPython.org Java give you jitters? Relax with wxPython!

After the extension C++ routine is called no python is called before
it ends and returns an integer. It is just a C++ program where I have
wrapped the main function using SWIG. If what you describe causes the
problem how do I check the GIL is released and release if it is not
(using SWIG).

Thanks,
Kasper

···

On Thu, Sep 25, 2008 at 11:49 PM, Robin Dunn <robin@alldunn.com> wrote:

Christopher Barker wrote:

Kasper Munch wrote:

When you say python API, do you mean wxPython? - And what is GIL?

sorry -- I mean calling back into the python interpreter at all. GIL is
the Global Interpreter Lock -- there are a number of python operations (I
can't tell you which!) that require the interpreter to lock so that multiple
threads don't et all hung up on each-other -- it can kill multi-threaded
performance.

Basically it boils down to this: Only one thread can be executing Python
byte-code or manipulating Python objects in any other way at one time. So
if you make a call into an extension module and it does not release the GIL
then *no* other Python code (including most Python C APIs that can result in
Python code being executed or Python objects being created) can be executed
until that extension call returns.

wxPython releases the GIL on the way in to most[1] of the wx wrappers,
reacquires it on the way back out, and also acquires it before calling event
handlers and other callbacks. Kasper, if your extension module does not
release the GIL then it could behave like what you have been describing.

[1] Except for lightweight things like dealing with wx.Point objects, etc.

--
Robin Dunn
Software Craftsman
http://wxPython.org Java give you jitters? Relax with wxPython!

_______________________________________________
wxpython-users mailing list
wxpython-users@lists.wxwidgets.org
http://lists.wxwidgets.org/mailman/listinfo/wxpython-users

--
Department of Integrative Biology
University of California, Berkeley
Office phone: (510) 643-6299

Kasper Munch wrote:

After the extension C++ routine is called no python is called before
it ends and returns an integer. It is just a C++ program where I have
wrapped the main function using SWIG. If what you describe causes the
problem how do I check the GIL is released and release if it is not

http://docs.python.org/api/threads.html

(using SWIG).

In SWIG you can use the %exception directive to wrap custom code around the call to the function being wrapped.

···

--
Robin Dunn
Software Craftsman
http://wxPython.org Java give you jitters? Relax with wxPython!