threads and unwanted recursive SafeYield call

Perhaps someone could offer some help on a problem I'm having with
threads. I have a user interface which has long-running processes of
two kinds. One kind of long-running process runs locally in the gui
thread; the other kind makes calls to web services to request
computation on a server. I would like both kinds of processes to be
able to output text to a wx.richtext.RichTextCtrl while they are
running. To complicate matters, the web services should be run in
parallel. Further, the text is all output using the logging module.
I have a custom handler whose emit function calls a function that puts
text into the RichTextControl. That function is a method of my main
frame's class which I call addMsg.

Ignoring the multiple threads for the moment, everything works fine.
I call one of the logger functions (e.g. logger.info('Some message')),
and that calls mainFrame.addMsg, and "Some message" appears in the
RichTextControl. Because mainFrame.addMsg calls wx.SafeYield, the
text appears immediately even if the long-running process is still
running.

Now comes the problem. I modified mainFrame.addMsg so instead of
putting the text in the RichTextControl, it posts a custom event. The
event is bound to mainFrame._addMsg (note the underscore), which does
the work of putting the text in the RichTextControl. To allow the
text to appear immediately, it is now _addMsg which calls
wx.SafeYield.

In other words...

Start long-running process.
Long-running process logs a message.
The logger calls addMsg.
addMsg posts an event.
The event triggers _addMsg.
_addMsg calls wx.SafeYield.

This all still works when the logger is called from the gui thread,
but when I call the logger from another thread, I get this message
over and over:

Traceback (most recent call last):
  File "C:\python\BMS-PDB\dcMainFrame.py", line 2100, in _addMsg
    wx.SafeYield()
  File "C:\python24\lib\site-packages\wx-2.8-msw-unicode\wx\_core.py",
line 7652, in SafeYield
    return _core_.SafeYield(*args, **kwargs)
wx._core.PyAssertionError: C++ assertion "wxAssertFailure" failed at ..
\..\src\msw\app.cpp(695) in wxApp::Yield(): wxYield called recursively

I tried replacing the wx.SafeYield call with a call to a function
"mySafeYield" that checks the stack and only calls SafeYield if there
isn't a SafeYield or Yield higher in the stack. (Yes, I know this is
ugly.) That just behaves as though I didn't call SafeYield. The text
doesn't appear right away, and the application hangs.

Here's the stack for a logger call from the gui thread when everything
is fine:

5: ?
4: main
3: MainLoop
2: MainLoop
1: _addMsg
0: mySafeYield
Call wx.SafeYield()

And here's one from another thread which looks for and skips the
second call:

8: ?
7: main
6: MainLoop
5: MainLoop
4: _addMsg
3: mySafeYield
2: SafeYield
1: _addMsg
0: mySafeYield
Detected recursive call at depth 2, do not call again.

This stack trace makes no sense to me. Why would SafeYield wind up
calling _addMsg? Is it re-posting the event?

That's probably a lot to follow, but any thoughts on why SafeYield is
recursive or a better approach would be greatly appreciated.

Matthew

MHC wrote:

Here's the stack for a logger call from the gui thread when everything
is fine:

5: ?
4: main
3: MainLoop
2: MainLoop
1: _addMsg
0: mySafeYield
Call wx.SafeYield()

And here's one from another thread which looks for and skips the
second call:

8: ?
7: main
6: MainLoop
5: MainLoop
4: _addMsg
3: mySafeYield
2: SafeYield
1: _addMsg
0: mySafeYield
Detected recursive call at depth 2, do not call again.

This stack trace makes no sense to me. Why would SafeYield wind up
calling _addMsg? Is it re-posting the event?

No, it's processing the next one in the pending event queue.

That's probably a lot to follow, but any thoughts on why SafeYield is
recursive or a better approach would be greatly appreciated.

Keep in mind that the yield functions are essentially starting a nested event loop that runs until there are no more events and then the yield function returns. So what is happening is that you are in a nested event loop and then one of the event handlers is called that calls yield again, which starts another nested event loop, and so on... Instead of wx.SaefYield try calling wx.GetApp().Yield(onlyIfNeeded=True)

OTOH, if you can avoid using any of the the yield functions you will be much better off. Is there a reason that the first long running task must be in the GUI thread? If moving it to a worker thread is too hard can it be broken into smaller chunks of work and then run each chunk in a series of EVT_IDLE handlers?

If the only reason you are calling yield is so the text widget will get painted then perhaps just calling its Update() method will be enough.

···

--
Robin Dunn
Software Craftsman