multiprocessing vs threading vs wx.CallAfter

wxPython Users
1

Hi all,

it seems that multiprocessing does not work alongside
wx.CallAfter() ? Has anyone else noticed that ?

  python-wxgtk3.0:
    Installiert: 3.0.2.0+dfsg-3
    Installationskandidat: 3.0.2.0+dfsg-3

I am doing the following: The user selects an item on the
list, the event handler kicks of a multiprocessing.Process()
which does a fairly lengthy transformation (say, retrieving
metadata from a large file). Eventually, the processing is
done and a callback is invoked with the results. Said
callback has been passed in when creating the Process().

The callback is called just fine. Results are delivered.
However, the callback then proceeds to update a TextCtrl with
help of

  wx.CallAfter(tctrl.SetValue, value)

  (because we are coming from another thread so need to
  switch over to the main, GUI, thread)

but neither that nor doing

  tctr.SetValue(value)

directly (expected to fail) does actually update the tctrl.

The very same code updates the ctrl just fine when replacing
the multiprocessing.Process() with threading.Thread() which
(intentionally) has got the same API.

Now, technically the difference is that multiprocessing uses
"subprocesses" while threading uses "real threads".
Supposedly the former are preferable because of the GIL (?).

SO, really, the question is -- is it a known fact that
wx.CallAfter doesn't seem to work when invoked from a
subprocess rather than a thread ?

Thanks,
Karsten

···

--
GPG key ID E4071346 @ eu.pool.sks-keyservers.net
E167 67FD A291 2BEA 73BD 4537 78B9 A9F9 E407 1346

2

I am doing the following: The user selects an item on the list

*a* wxPython ListCtrl (which doesn't matter)

, the event handler kicks of

kicks *off*, sorry

  wx.CallAfter(tctrl.SetValue, value)

  (because we are coming from another thread so need to
  switch over to the main, GUI, thread)

but neither that nor doing

  tctr.SetValue(value)

Just another typo, the problem is not the missing 'l'.

Karsten

···

On Sun, Jan 29, 2017 at 04:12:23PM +0100, Karsten Hilbert wrote:
--
GPG key ID E4071346 @ eu.pool.sks-keyservers.net
E167 67FD A291 2BEA 73BD 4537 78B9 A9F9 E407 1346

3

Personally, I’d be surprised if CallAfter DID work from another process. It is designed to be thread safe, but that’s sot easier. If you want to run other processes, you’ll need some inter-process communication of some sort.

···

But if you only need to know that the process is done – doesn’t multiprocess tell you that? And you can use CallAfter() from the thread that kicks off the process?

-CHB

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@noaa.gov

On Jan 29, 2017, at 10:07 AM, Karsten Hilbert Karsten.Hilbert@gmx.net wrote:

On Sun, Jan 29, 2017 at 04:12:23PM +0100, Karsten Hilbert wrote:

I am doing the following: The user selects an item on the list

a wxPython ListCtrl (which doesn’t matter)

, the event handler kicks of

kicks off, sorry

wx.CallAfter(tctrl.SetValue, value)

(because we are coming from another thread so need to
switch over to the main, GUI, thread)

but neither that nor doing

tctr.SetValue(value)

Just another typo, the problem is not the missing ‘l’.

Karsten

GPG key ID E4071346 @ eu.pool.sks-keyservers.net
E167 67FD A291 2BEA 73BD 4537 78B9 A9F9 E407 1346

–
You received this message because you are subscribed to the Google Groups “wxPython-users” group.
To unsubscribe from this group and stop receiving emails from it, send an email to wxpython-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

4

Maybe you can call join() from another thread? And then call CallAfter() from that thread?

···

https://pymotw.com/2/multiprocessing/basics.html

CHB

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@noaa.gov

On Jan 29, 2017, at 10:07 AM, Karsten Hilbert Karsten.Hilbert@gmx.net wrote:

On Sun, Jan 29, 2017 at 04:12:23PM +0100, Karsten Hilbert wrote:

I am doing the following: The user selects an item on the list

a wxPython ListCtrl (which doesn’t matter)

, the event handler kicks of

kicks off, sorry

wx.CallAfter(tctrl.SetValue, value)

(because we are coming from another thread so need to
switch over to the main, GUI, thread)

but neither that nor doing

tctr.SetValue(value)

Just another typo, the problem is not the missing ‘l’.

Karsten

GPG key ID E4071346 @ eu.pool.sks-keyservers.net
E167 67FD A291 2BEA 73BD 4537 78B9 A9F9 E407 1346

–
You received this message because you are subscribed to the Google Groups “wxPython-users” group.
To unsubscribe from this group and stop receiving emails from it, send an email to wxpython-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

5

Basically you are trying to perform GUI operations, e.g. tctrl.SetValue,
from another thread - this is not recommended full stop. You need to
provide an inter-process communications mechanism and let the GUI thread
do all GUI operations.

One such mechanism is to have the thread that you need to set the value
raise a custom event and have a handler for that same event in the GUI
thread.

···

On 29/01/2017 18:07, Karsten Hilbert wrote:

On Sun, Jan 29, 2017 at 04:12:23PM +0100, Karsten Hilbert wrote:

I am doing the following: The user selects an item on the list

*a* wxPython ListCtrl (which doesn't matter)

, the event handler kicks of

kicks *off*, sorry

  wx.CallAfter(tctrl.SetValue, value)

  (because we are coming from another thread so need to
  switch over to the main, GUI, thread)

but neither that nor doing

  tctr.SetValue(value)

Just another typo, the problem is not the missing 'l'.

Karsten

--
Steve (Gadget) Barnes
Any opinions in this message are my personal opinions and do not reflect
those of my employer.

6

Basically you are trying to perform GUI operations, e.g. tctrl.SetValue,
from another thread - this is not recommended full stop.

Not that I think:

You need to provide an inter-process communications mechanism and let the GUI thread
do all GUI operations.

Exactly: wx.CallAfter() which is, to my knowledge, doing
exactly that: send a custom event to "itself" that being
processing in the main, GUI, thread.

What am I missing ?

Thanks for you input,
Karsten

···

On Mon, Jan 30, 2017 at 07:40:36AM +0000, Steve Barnes wrote:
--
GPG key ID E4071346 @ eu.pool.sks-keyservers.net
E167 67FD A291 2BEA 73BD 4537 78B9 A9F9 E407 1346

7

The wxPython wiki

  CallAfter - wxPyWiki

expressly sayeth so:

  wx.CallAfter takes a function and its arguments, and
  calls the function when the current event handler has
  exited. It is a safe way of requesting action of a Window
  from another thread.

Now, it is not that I am unable to this working - it does,
very well, and has for years in this application (GNUmed). It
is only that when I read up on threading.Thread() that I
noticed it now refers to a newer, better designed, "more
lightweight" (?) multiprocessing module so I thought to
myself maybe I should start using that.

So once I implemented the delayed GUI thread invocation with
multiprocessing it stopped working. The change back to
threading is but ONE line, namely from:

  worker = multiprocessing.Process(...)

to

  worker = threading.Thread(...)

since the API is (nearly, no .pid in threading) the same.

"threading" works fine, "multiprocessing" does not (more
precisely, multiprocessing does everything perfectly _except_
deliver the wx.CallAfter to the GUI thread). It must be
somewhat different under the hood, as the documentation says:

  The multiprocessing package offers both local and remote
  concurrency, effectively side-stepping the Global
  Interpreter Lock by using subprocesses instead of
  threads.

This

  multithreading - Multiprocessing vs Threading Python - Stack Overflow

gives a bit more insight making me think that maybe
wx.CallAfter() isn't designed to be used across processes
(Wiki says "from another *thread*" - not "*process*").

Which was my initial question :slight_smile:

Can anyone explain or confirm this more clearly ?

Thanks,
Karsten

···

On Mon, Jan 30, 2017 at 10:36:37AM +0100, Karsten Hilbert wrote:

Exactly: wx.CallAfter() which is, to my knowledge, doing
exactly that: send a custom event to "itself" that being
processing in the main, GUI, thread.

--
GPG key ID E4071346 @ eu.pool.sks-keyservers.net
E167 67FD A291 2BEA 73BD 4537 78B9 A9F9 E407 1346

8

:slight_smile:

Steve, I was missing your *precise* wording: You say "from
another *thread* - this is not recommended". That much is
clear. And I don't do it either. But then you say: "provide
an inter-*process* communications mechanism" which talks
about *processes* (which is what multiprocessing uses) rather
than *threads*. So, yes, while the first was misleading me
into thinking you were referring to inter-THREAD
communications in the second sentence you did refer to
inter-PROCESS - which seems correct, after all, because
wx.CallAfter() does not seem to be inter-process (but
inter-thread).

Karsten

···

On Mon, Jan 30, 2017 at 10:36:37AM +0100, Karsten Hilbert wrote:

> Basically you are trying to perform GUI operations, e.g. tctrl.SetValue,
> from another thread - this is not recommended full stop.

Not that I think:

> You need to provide an inter-process communications mechanism and let the GUI thread
> do all GUI operations.

Exactly: wx.CallAfter() which is, to my knowledge, doing
exactly that: send a custom event to "itself" that being
processing in the main, GUI, thread.

What am I missing ?

--
GPG key ID E4071346 @ eu.pool.sks-keyservers.net
E167 67FD A291 2BEA 73BD 4537 78B9 A9F9 E407 1346

9

Karsten Hilbert wrote:

Can anyone explain or confirm this more clearly ?

wx.CallAfter doesn't work from a subprocess, this you've discovered and
as others have pointed out, it would be pretty surprising if it did.
Running code in a separate memory space, i.e. in a child process, is
different from running it in a child thread of the same process. AFAIK,
sharing code objects across processes means pickling and unpickling
them, so it's unlikely that this would just work smoothly with custom
wxPython event objects as created by wx.CallAfter, or even that
wx.CallAfter could find the wx.App instance to post the created event to.

You'll need to handle updating the GUI, whether or not that includes a
call to CallAfter or whatever else, in the main process, and implement
some other way to be notified when your child process has finished doing
whatever it's doing. The API for multiprocessing and threading may be
similar, but in this instance that fact has lead you to also think that
they are similar in their implementation.

HTH.

···

--
James Scholes
http://twitter.com/JamesScholes

10

wx.CallAfter doesn't work from a subprocess, this you've discovered and
as others have pointed out, it would be pretty surprising if it did.
Running code in a separate memory space, i.e. in a child process, is
different from running it in a child thread of the same process. AFAIK,
sharing code objects across processes means pickling and unpickling
them, so it's unlikely that this would just work smoothly with custom
wxPython event objects as created by wx.CallAfter, or even that
wx.CallAfter could find the wx.App instance to post the created event to.

Thanks for the confirmation.

You'll need to handle updating the GUI, whether or not that includes a
call to CallAfter or whatever else, in the main process, and implement
some other way to be notified when your child process has finished doing
whatever it's doing. The API for multiprocessing and threading may be
similar, but in this instance that fact has lead you to also think that
they are similar in their implementation.

Likely so. I have resolved on using threading as I used to
and which does work for the reasons you mention.

Thanks to all for the input,
Karsten

···

On Mon, Jan 30, 2017 at 10:45:12AM +0000, James Scholes wrote:
--
GPG key ID E4071346 @ eu.pool.sks-keyservers.net
E167 67FD A291 2BEA 73BD 4537 78B9 A9F9 E407 1346

11

Sorry for the confusion Karsen - the thing is that you have a hierarchy
of method structures:

  - Single Threaded: Simple, resource limited, always on one core on one
machine. Always works, (unless it runs out of resources), but when
things get busy pauses everything else it is doing, such as updating the
GUI.
  - Multi-Threaded: Multiple threads running on a single core on one
machine, slightly less constrained for resources, will work if all of
the methods are thread safe, (will sometimes work some of the time even
if they are not). Since the memory space is shared data can be passed
very quickly but you are still restricted to a single cores resources.
GUI doesn't freeze when things get busy but might get slow - use
wxCallAfter for GUI updates at the next opportunity.
  - Multi-Process: Multiple Processes running on Multiple cores, with
possibly separate resources, possibly each running multiple threads, one
such thread is the GUI. Use Inter-Process communications, (usually
restricted to pickle-able items), to pass messages &/or data between
process - usually one process will have the GUI thread plus a message
listener thread which will use wxCallAfter to update the GUI when it
receives an interprocess message.
  - Distributed-Processing: This is an extension of Multi-Processing
with specifically an Inter Process Communication mechanism that can
transport information between machines as well as between cores. The
most complex as you need to deal with possible transmission delays, data
loss, communications or remote node being lost, load balancing, etc.
But: potentially virtually unlimited resources, possibly multiple GUI in
multiple locations, for large data or complex calculations is hard to beat.

Some more reading:

···

On 30/01/2017 10:06, Karsten Hilbert wrote:

On Mon, Jan 30, 2017 at 10:36:37AM +0100, Karsten Hilbert wrote:

Basically you are trying to perform GUI operations, e.g. tctrl.SetValue,
from another thread - this is not recommended full stop.

Not that I think:

You need to provide an inter-process communications mechanism and let the GUI thread
do all GUI operations.

Exactly: wx.CallAfter() which is, to my knowledge, doing
exactly that: send a custom event to "itself" that being
processing in the main, GUI, thread.

What am I missing ?

:slight_smile:

Steve, I was missing your *precise* wording: You say "from
another *thread* - this is not recommended". That much is
clear. And I don't do it either. But then you say: "provide
an inter-*process* communications mechanism" which talks
about *processes* (which is what multiprocessing uses) rather
than *threads*. So, yes, while the first was misleading me
into thinking you were referring to inter-THREAD
communications in the second sentence you did refer to
inter-PROCESS - which seems correct, after all, because
wx.CallAfter() does not seem to be inter-process (but
inter-thread).

Karsten

  -

  -

Hope that start to make things clearer.
--
Steve (Gadget) Barnes
Any opinions in this message are my personal opinions and do not reflect
those of my employer.

12

a few additional notes / comments:

(I’m not sure if Steve Barnes meant this to be general or python - specific, but I’m writing from a python-specific perspective)

···
  • Multi-Threaded: Multiple threads running on a single core on one

machine, slightly less constrained for resources, will work if all of

the methods are thread safe, (will sometimes work some of the time even

if they are not). Since the memory space is shared data can be passed

very quickly but you are still restricted to a single cores resources.

python uses the OS multi-threading lib, and is not restricted to one core. However, the GIL does assure that only one thread is actually doing anything at once, so while different threads may be on different cores, you don’t get “real” parallel processing.

GUI doesn’t freeze when things get busy but might get slow - use

wxCallAfter for GUI updates at the next opportunity.

In short, multi-threading does not help if you want multiple computationally expensive code running at the same time. But it does help a lot if you have multiple threads that are spending a fair bit of time waiting on other resources (network, database, files system), and can allow your GUI to be far more responsive if you have ONE computationally expensive thread.

  • Multi-Process: Multiple Processes running on Multiple cores, with

possibly separate resources, possibly each running multiple threads, one

such thread is the GUI. Use Inter-Process communications, (usually

restricted to pickle-able items), to pass messages &/or data between

process - usually one process will have the GUI thread plus a message

listener thread which will use wxCallAfter to update the GUI when it

receives an interprocess message.

Exactly – much more robust in the face of computationally intensive work, but heavier weight and more complex.

Karsten,

Do you have any performance issues with you threaded code? If not, then you’re done :slight_smile:

If so, then maybe multi-processing could help.

-CHB

–

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@noaa.gov

13

...

Karsten,

Do you have any performance issues with you threaded code? If not, then
you're done :slight_smile:

Short and to the point :-))

No, I don't. So I'm done. I just wanted to confirm that my
finding was correct.

Thanks Chris and Steve for spelling out the details !

Karsten

···

On Mon, Jan 30, 2017 at 01:24:43PM -0800, Chris Barker wrote:
--
GPG key ID E4071346 @ eu.pool.sks-keyservers.net
E167 67FD A291 2BEA 73BD 4537 78B9 A9F9 E407 1346

14

I had a similar problem. I needed to perform a lengthy operation without locking the GUI, and also update a progress bar as the lengthy operation progressed. After long research I found the solution: use threading + wx.CallAfter + PubSub module.

···

On Sunday, January 29, 2017 at 8:28:36 PM UTC+3, Karsten Hilbert wrote:

Hi all,

it seems that multiprocessing does not work alongside

wx.CallAfter() ? Has anyone else noticed that ?

    python-wxgtk3.0:

      Installiert:           3.0.2.0+dfsg-3

      Installationskandidat: 3.0.2.0+dfsg-3

I am doing the following: The user selects an item on the

list, the event handler kicks of a multiprocessing.Process()

which does a fairly lengthy transformation (say, retrieving

metadata from a large file). Eventually, the processing is

done and a callback is invoked with the results. Said

callback has been passed in when creating the Process().

The callback is called just fine. Results are delivered.

However, the callback then proceeds to update a TextCtrl with

help of

    wx.CallAfter(tctrl.SetValue, value)



    (because we are coming from another thread so need to

    switch over to the main, GUI, thread)

but neither that nor doing

    tctr.SetValue(value)

directly (expected to fail) does actually update the tctrl.

The very same code updates the ctrl just fine when replacing

the multiprocessing.Process() with threading.Thread() which

(intentionally) has got the same API.

Now, technically the difference is that multiprocessing uses

“subprocesses” while threading uses “real threads”.

Supposedly the former are preferable because of the GIL (?).

SO, really, the question is – is it a known fact that

wx.CallAfter doesn’t seem to work when invoked from a

subprocess rather than a thread ?

Thanks,

Karsten

–
GPG key ID E4071346 @ eu.pool.sks-keyservers.net

E167 67FD A291 2BEA 73BD 4537 78B9 A9F9 E407 1346

15

Yeah, that would work.

Thanks for following up.

Karsten

···

On Sun, Feb 05, 2017 at 12:17:29PM -0800, steve wrote:

I had a similar problem. I needed to perform a lengthy operation without
locking the GUI, and also update a progress bar as the lengthy operation
progressed. After long research I found the solution: use *threading +
wx.CallAfter + PubSub module.*

--
GPG key ID E4071346 @ eu.pool.sks-keyservers.net
E167 67FD A291 2BEA 73BD 4537 78B9 A9F9 E407 1346