Newbie: how do I start a Python process that runs in parallel to the GUI event loop?

Newbie Help
#1

I am trying to add a GUI to a Python program that controls a measurement instrument. When the program is started, it automatically connects and configures the instrument. This process takes many seconds or a few minutes, so it must not be handled within the event loop of the new GUI. The process prints a bunch of messages that need to be shown in the GUI window.

So far, I put the event loop of the wxPython GUI at the core of my program. Now I need to figure out the following:
(1) How do I start the process to connect/configure the instrument outside of the event loop?
(2) How do I redirect the process messages to the GUI (I found this, but I don’t know if it’s the right way to go: https://www.blog.pythonlibrary.org/2009/01/01/wxpython-redirecting-stdout-stderr )

As a newbie to wxWidgets I am not sure where to look. Some pointers or a (pseudo) example would be great!

Thanks!

#2

You probably want to use threads.
See e.g. here:

Regards,
Dietmar

#3

I’ve had success using the multiprocessing module to start a new process. To communicate back and forth from the GUI to the worker I create a few Queues (the multiprocessing module has its own Queue implementation). You just need to set a timer in your GUI to check the queues for incoming messages. You can use messages in a Queue from the GUI to the process to issue commands, and in the end to tell it to die.
The tricky part for me was to find a way to interrupt the process if it is off in the weeds. The answer was to use the signal module and create a signal handler that looks for, say, signal.SIGUSR1 and raises a KeyboardInterrupt exception in the process. Then surround your process with a try: clause and catch that exception. Then your GUI can issue an os.kill(proc_pid, signal.SIGUSR1) to stop the process. You get the pid from the return value of the mp.Process(…) call that started the process.

#4

Thanks to the lockdown I had some time to cook up an example. Here’s the main program:
mp_example.py (2.2 KB)
and here’s the simple process:
my_process.py (672 Bytes)
All the process does is run a timer and report back.

#5

My take on this issue is using a multiprocess and a thread.

Create your wxApp. The App will then define a wxTimer and couple of queues and start a multiprocess. The multiprocess will then run a thread that will act as a messenger to communicate through the queues with you wxApp. When closing the wxApp you will just need to stop the wxTimer, send a message asking the thread to stop and then terminate the process.

It’s a lot easier than it seems actually. I wrote a thermometer display for Raspberry Pi that does exactly that: the sensor reading runs in a different process than the wxApp’s. Take a look at https://www.tacao.com.br/digitherm.html .

#6

I’m a bit surprised about the focus on interrupting hanging programs. I’m using GPIB, USB, Ethernet, RS232 and some other buses and always was fine using timeouts.

The performance should be higher when you use a separate process.

With threads on Windows, the performance can be increased a bit by setting the thread performance:
win32process.SetThreadPriority(win32api.GetCurrentThread(), win32process.THREAD_PRIORITY_TIME_CRITICAL)
Using this, I could get 5 acquisitions per second from some GPIB instruments in an application with embedded matplotlib canvases.

#7

Well, as I wrote, I am a complete newbie, if not a noob… I am trying to understand (in the context of my application), what is the difference between “threads” and “processes”? Why should I use one or the other?

#8

Well, a thread is some kind of parallelism within a program.
A process is a second program which will then be controlled from the first.
In both cases, the threads or processes need to communicate with each other, but all in all, threads should be easier to implement and debug. The above wiki article LongRunningTasks should get you started. The second link to another discussion thread is about using wx.CallAfter instead of the more complicated communication methods. As you wrote that just the initialization is taking a lot of time, it might be enough for you to:

  • start the thread for the initialization
  • when this is done, tell this to the main thread using e.g. wx.CallAfter(self.main_program.initialization_done)
  • do the rest of the IO inside the main thread

About the redirection of stdout/stderr: don’t do this. Use proper communication.

Regards,
Dietmar

#9

The initialisation of the instrument prints some information using print(…) statements, which should be displayed to the user. This is why I want to “redirect” the output of the print(…) statements to the GUI in some way. What is wrong with redirecting STDOUT to the GUI, and what would be a better way to do this?

#10

Using print for some debug messages is OK. Usually I run all my wxPython programs with a console window to see these, as well as unhandled exceptions.

But for communication, use proper function calls.

E.g. supply a callback function to your thread and then the thread will call this with status informations using wx.CallAfter. The main thread will update it’s controls accordingly. Or use custom events to communicate. The wiki article should answer your questions. The details depend on your specific needs.

#11

I get your point – but it seems that avoiding the print(…) logging from the instrument control code would mean I’d have to rewrite this code for the GUI version of my program. I would have to maintain this code in parallel to the terminal/console version. This is asking to for too much trouble, and I will not go down this hole.
What is the best way to show print(…) logging messages from the instrument control in the GUI?
What’s the problem with the “redirect” method described here: https://www.blog.pythonlibrary.org/2009/01/01/wxpython-redirecting-stdout-stderr ?

#12

you don’t have to dive into the signal module…
mp_example_evt.py (2.1 KB) my_process_evt.py (525 Bytes)

#13

In the particular use case in which I needed to use signal, the process inside my while loop was a call into third party libraries connected to hardware that would hang from time to time, so the abort_event.is_set() call was never made to exit the while loop. The keyboard_interrupt would make it exit though.

#14

Well, I’m on a Windows and in your example I could take all the signal stuff off (except in the ‘os.kill’) and it still worked, i.e. the exception in the target process was useless: the os.kill killed the process unconditionally, which is pretty rough! I suppose some timeout in that called library would be useful

#15

Yes, sorry, I don’t do Windows :wink:
And your method for normal shutdown is much better. If things aren’t locked up then when you catch the shutdown event then you can close files, shut off hardware, etc.

#16

I believe I have worked it out – thanks to everyone for your help!

(1) Once I worked out how threads are part of the GUI frame, I made a thread to talk to the instrument.
(2) Once I realised that events can carry data (for example log messages) I made the GUI logging work by overloading the logging method provided by the instrument classes. The overloaded method sends an “logging event” with the log message attached to the event, which is then handled / displayed from within the GUI event loop. Figuring out the event.data thing was maybe the hardest part for me, because my 25 year old experience in programming event loops with Pascal or C didn’t leave any room for the even.data concept in my brain…

#17

I suppose you looked at ‘wx.lib.newevent’, or have you found something else about this ‘event.data’?

#18

In my first post above, I pasted the link to another thread, as the simplest alternative is to use wx.CallAfter.
With CallAfter you don’t need to worry about events, (thread aware) queues etc.

Sometimes it’s the easiest to follow the first reply.

#19

You are right. Using wx.CallAfter(…) within the thread to call a method in the GUI handler is easier and more direct, as you don’t need to set up any event definitions. I didn’t see the wood for the trees.

#20

and you don’t have to ‘queue’ anymore…
mp_example_sm.py (2.1 KB) my_process_sm.py (396 Bytes)