Running multiple `MainLoop`s in separate threads

I want to design a program which watches replies from one or more servers and then do something with these replies. This ‘something’ should be implemented as plugins. These plugins just receive the server’s replies and for example display them in the terminal or play a sound file. I also want to write a plugin, which displays this info in a wx window. This is, where the problems start (actually they start in the moment when I want to cleanly shut down these wx windows at program closure).

Let me explain the design of the program before showing the minimal use case at the end of this message:

Since waiting for the server replies is a blocking operation – and I don’t want to renew this every second or so –, I implemented this part in a daemonized thread (the BlockingThread in my minimal use case). Before watching the replies from the server, it handles the loading of the plugins (MyPlugin and WxPlugin). The plugins are responsible to do something with the information received (not implemented in my minimal use case) and different plugins can be used for every thread watching the server. Because these threads are daemonized, they are automatically closed when the program stops. (The server does not mind this.)

All this is implemented with the help of a controlling thread (ControlThread), which acquires a lock and waits for it to be released with the help of join().

Any plugin can request a shutdown of the entire program by calling the exit() method of the underlying BlockingThread, which then calls the ControlThread’s exit() method. This method releases the acquired lock and the main program proceeds and would delete all blocking threads at its end. But before all daemonized threads get killed, they get the chance to clean up, because their stop() method is called. Sequentially the BlockingThread calls the stop() method of each plugin loaded for it. So that the plugin can also clean up.

This works as expected.

Mind you, maybe all this is overcomplicated and there are better ways to do it?

But the plugin displaying the message in a wx window is somewhat special, because that wx window needs to run a MainLoop. The plugin WxPlugin runs this main loop in a separate daemonized thread.

Up to here, everything works as expected and the wx windows can display the information received from the server (not implemented in my minimal use case).

The problem occurs, when I want to close the whole lot e.g. by clicking on any of the wx windows. I bind the mouse click to a method, which calls the underlying plugin’s exit() method, thus initiated the shutdown procedure described above.

When in return to the shutdown request the plugin’s stop() method gets called, it Destroy()s the wx frame first and then calls the app’s ExitMainLoop(). This is done for all WxPlugins. At the very end I sometimes get error messages or segmentation faults and sometimes I don’t get any error. These are the errors I received (always after the second WxPlugin.stop() method had finished):

  • Gdk-CRITICAL **: 19:30:55.260: _gdk_frame_clock_freeze: assertion 'GDK_IS_FRAME_CLOCK (clock)' failed
  • Segmentation fault (core dumped)

I checked with IsMainLoopRunning() a second after calling ExitMainLoop(). I noticed, that

  1. the main loop of the first BlockingThread is still running after ExitMainLoop() and
  2. the main loop of the second BlockingThread is never running after ExitMainLoop()!

I also checked with id(), if the main loop might be the same for both BlockingThreads, but they are different objects (as I would have expected).

How can I cleanly shut down all the wx main loops and their associated windows?

Now the minimal use case:

import threading
import time

import wx


class MyFrame(wx.Frame):

    def __init__(self, parent, exit_callback, title, id=wx.ID_ANY):
        super().__init__(parent, id=id)
        self.SetTitle(title)

        self.exit_callback = exit_callback

        self.Bind(wx.EVT_LEFT_DOWN, self.OnMouseLeft)
        self.Show()

    def OnMouseLeft(self, event):
        print('MyFrame.OnMouseLeft()')
        self.exit_callback()


class MyPlugin:

    def __init__(self, exit_callback, title):
        print(f'{title} - MyPlugin()')
        super().__init__()

        self.exit_callback = exit_callback
        self.title = title

    def stop(self):
        print(f'{self.title} - MyPlugin.stop()')

    def exit(self):
        print(f'{self.title} - MyPlugin.exit()')
        self.exit_callback()


class WxPlugin:

    def __init__(self, exit_callback, title):
        print(f'{title} - WxPlugin()')
        super().__init__()

        self.exit_callback = exit_callback
        self.title = title

        self.app = wx.App()
        self.frame = MyFrame(None, self.exit, title)
        self.app.SetTopWindow(self.frame)

        self.t = threading.Thread(target=self.app.MainLoop)
        self.t.daemon = True
        self.t.start()

    def stop(self):
        print(f'{self.title} - WxPlugin.stop()')
        self.app.ExitMainLoop()
        print(f'{self.title} - MainLoop stopped')
        time.sleep(1)
        print(f'{self.title} - {self.app.IsMainLoopRunning()=}')

    def exit(self):
        print(f'{self.title} - WxPlugin.exit()')
        self.exit_callback()

class BlockingThread(threading.Thread):

    def __init__(self, exit_callback, title):
        super().__init__()

        self.exit_callback = exit_callback
        self.title = title
        self.plugins = [
            MyPlugin(self.exit, title),
            WxPlugin(self.exit, title),
        ]

    def run(self):
        while True:
            time.sleep(30 * 60)  # Blocking 30 minutes

    def stop(self):
        print(f'{self.title} - BlockingThread.stop()')
        for plugin in self.plugins:
            plugin.stop()

    def exit(self):
        print(f'{self.title} - BlockingThread.exit()')
        self.exit_callback()


class ControlThread(threading.Thread):

    def __init__(self):
        super().__init__()
        self._lock = threading.Lock()
        self._lock.acquire()

    def run(self):
        print('ControlThread.run()')
        self._lock.acquire()

    def exit(self):
        print('ControlThread.exit()')
        self._lock.release()


if __name__ == '__main__':
    ctrl = ControlThread()
    ctrl.start()

    threads = [
        BlockingThread(ctrl.exit, 'Thread 1'),
        BlockingThread(ctrl.exit, 'Thread 2'),
    ]
    for t in threads:
        t.daemon = True
        t.start()

    ctrl.join()

    for t in threads:
        t.stop()

well, one wx App and the ‘workers’ are threaded (have a look at this) :wink:

If I understand you (and the thread) correctly (please correct me, if that’s wrong), you suggest to run the main loop in the main program and the watcher threads in separate threads communicating with the main loop. This certainly works, but makes the whole program depending on wx, whereas at the moment it would be possible to run the program without wx installed, if none of the plugins request it! That’s a feature I want to keep.

well, I run your snippet and it doesn’t do anything at all even with wx installed (may be it is corrupted?) :thinking:

whereas at the moment it would be possible to run the program without wx installed

That comment referred to the plugin system, which is not implemented in the minimal use case shown.

I noticed, that in my browser the minimal use case is cut off before it ends and thus cannot be selected and copied in its whole glory. Can you check, that it has 125 lines and shows the if __name__ == '__main__': clause?

Using the copy icon at its top right corner works and copies the complete example.

When run, the example should show two empty windows, titled Thread 1 and Thread 2. Clicking on any of this shuts down the program – and often results in an error or a segmentation fault.

Testing using wxPython 4.2.2 gtk3 (phoenix) wxWidgets 3.2.6 + Python 3.12.3 + Linux Mint 22, I got critical errors in approximately 1 out of 8 runs of the example code.

The critical errors are not all the same. Here are 3 examples:

gtk_widget_get_toplevel: assertion 'GTK_IS_WIDGET (widget)' failed

(multiple_mainloops_in_separate_threads_1.py:45143): Gtk-CRITICAL **: 18:14:33.210: _gtk_widget_captured_event: assertion 'WIDGET_REALIZED_FOR_EVENT (widget, event)' failed
(multiple_mainloops_in_separate_threads_1.py:45435): GLib-CRITICAL **: 18:19:03.899: Source ID 21 was not found when attempting to remove it
(multiple_mainloops_in_separate_threads_1.py:45544): Gdk-CRITICAL **: 18:20:54.373: _gdk_frame_clock_freeze: assertion 'GDK_IS_FRAME_CLOCK (clock)' failed

Personally, I find the typical use case of a wxPython application running a single child thread at a time can be tricky at times.

However, I find this idea of having a hierarchy of objects spawning child threads to be really confusing and made my head hurt!

Yes, that’s what I also noticed: Sometimes I get errors and segmentation faults, sometimes not. Do you happen to know, why this is not consistent?

Yes, it does! I could not find a better way, how to kill a long running blocking input operation, but to put it in a daemonized thread. Any other ideas?

Having a plugin showing a wx window gave rise to run its main loop in a separate thread. If there is another way, I would appreciate to hear it!

The idea to send an exit request arose with the wx window. Without a gui, I could simply press Ctrl-C to terminate the program – this actually works without showing any errors.

The second idea to bubble up a close request arose from the observation that I get a segmentation fault, if I end the program with the simple ctrl.join() statement (just try it and comment out the last for t in threads: t.stop() loop!) So the idea was, to give each part the chance to close down properly. But this does not work either …

In other wxPython applications I have seen cases where events are still being triggered by child threads while wxPython and/or wxWidgets is in the process of terminating itself. It can even happen in applications with no child threads, for example when a timer is involved. In these cases some of the UI components will have been destroyed so that an error occurs when they can’t be accessed. However, I don’t know if that is what is happening in your example.

Sorry, but I don’t have any idea how to fix the problem.

What surprises me, is that the main loop is still running after calling .ExitMainLoop() in the plugin requested by the first BlockingThread, but the main loop is always stopped in the second BlockingThread. This can be seen in the output of line 62. Could this be the cause of the problem?

I tried to wait for the main loop to finish with a while self.app.IsMainLoopRunning(): time.sleep(0.1) instead of the `time.sleep(1) in line 61. But it never finishes!

How can I reliably stop the main loop in both instances of the WxPlugin?

In the documentation for wx.App, it says:

Every wx application must have a single wx.App instance

Your example creates 2 wx.App objects. Perhaps the problem is that having more than one wx.App object creates a conflict in the toolkit?

TLDR, but “Running multiple MainLoops in separate threads” is always the wrong answer.

Unfortunately I don’t know more recent documentation, so please have a look at this page:

https://wiki.wxpython.org/LongRunningTasks

Check the first code example for WorkerThread and PostEvent.

I have a use case with a a single background thread collecting data and the frontend displaying it.
For this I use self.data = collections.deque(). The worker thread appends data on one end, the GUI consumes from the other, triggered by a wx.Timer that fires every second.
For my use cases, that causes less performance overhead than posting events for each dataset.

If you are watching /many/ servers, asyncio might be the answer.

Regards,
Dietmar

Good Lord, that is what I pointed out in my first reply and it is written in the docs

with every of your lovely Plugins you create some sort of a new main loop, which is a sacrilege

BTW, my example I pointed out to you takes all your dead ends here into consideration
if you have noticed the first ten seconds there is no frame visible and that is the state if there is no wx Frame (but a GUI without frame will confuse the user) :rofl:

Oh my god, I completely forgot, that wx is not thread safe.

Just one comment on my design: I do not want to write a wx application, which watches a server (where all pointers given are more than valid), but an application which watches a servers and maybe displaying some information in wx windows (through a plugin). If the user decides that s/he does not need a fancy GUI window, s/he does not install the wx plugin and the program runs without having wxPython installed!

That’s why the watching thread is running the plugins with the wx.App and not the other way round.

Looks, like this is not so easy to achieve. Thanks all for your help.

It’s probably possible to run the mainloop in a thread that’s not the main thread.
You just need to make sure you don’t start multiple of them and interact with the GUI only in this thread. When the main window closes, the main loop will terminate. I have never tried re-starting it, but would expect it to work.

Other than that, I often write code that can be run in the console (by checking for __name__=="__main__") or be imported and controlled from a GUI application.

I don’t know if you have used the dynamic import yet
so at the start keep an instance in main thread (calling it wx) which just serves this dynamic import of your wx modules which are requested by the user via the ‘watching thread’ (how I don’t know since there is no Gui yet)
when wx is triggered it first loads an init module which loads wx itself and initiates the App, calls the wx application and starts the loop
if it is triggered again it just calls the wx application, no new App and loop
so far I think it should work :slightly_smiling_face: