Interrupting computation

wxPython Users
1

Guys,

I know this is a topic for the Python list, not the wxPython, but you guys are so ressourceful that I dare submitting my problem to you.
As in most of the software, an 'Cancel computation' button is required and I have trouble to make it work clean.

Here is how my application works. The Cancel computation button is, of course, in the main thread.

                          MainThread (mostly the GUI)
                               >
                       ComputationThread1 (threading.Thread)
      at the end of run(), wx.CallAfter(), starts the next thread
                               >
                       ComputationThread2 (threading.Thread)
                               >
                      StartProcesses (a regular class)
                      > > >
                     P1 P2 P3 (those are multiprocessing.Process objects)

     etc.

In StartProcesses, I define a multiprocessing.Manager object to collect the list of PIDs, what is done in each process using os.getpid(). StartProcesses has also the run() method that creates P1,P2,P3 and uses the join() method to make sure they are all terminated before going further.

Now, cancelling the computation: for processes, I get the Manager object from StartProcesses and kill each process using os.kill(pid, 3). Not sure the signal is the good one (I'm on Windows). For the thread, I turn a flag off so that it knows the computation is interrupted. Basically, it goes to wx.CallAfter() and there, instead of starting the next thread, I just stop the computation workflow.

After some 'hit my head against the nearest wall' sessions, it works more or less: when I cancel the computation, I get all processes (P1, ...) killed (at least, I hope, don't really know how to check), and only the MainThread is still alive (I check this by using threading.enumerate()).

Now the problem: if I look at Windows Task manager -> Processes, I have only one python.exe process when I start the application. When I launch the computation, I get another one, than a few more when it gets to running P1, P2, etc. When I cancel the computation, as all processes P1, etc. and the ComputationThread are terminated, I should get back to only one python process, but no, there are 2 of them! So, I'd like to know why there is an extra python process still running, and what it is ??? When I quit the application, everything is gone (at least), but now, if I do compute/cancel several times, without quitting the application, everytime one extra python process is added, so after 3 loops, for example, I have 4 python processes running.

Sorry, this is a bit complicated, but if anyone has a hint, that would be great.

Thanks,

Raphael

2

Hi,

Guys,

I know this is a topic for the Python list, not the wxPython, but you guys
are so ressourceful that I dare submitting my problem to you.
As in most of the software, an 'Cancel computation' button is required and I
have trouble to make it work clean.

Here is how my application works. The Cancel computation button is, of
course, in the main thread.

                    MainThread \(mostly the GUI\)
                         >
                 ComputationThread1 \(threading\.Thread\)
at the end of run\(\), wx\.CallAfter\(\), starts the next thread
                         >
                 ComputationThread2 \(threading\.Thread\)
                         >
                StartProcesses \(a regular class\)
                >        >        >
               P1       P2       P3  \(those are multiprocessing\.Process

objects)

etc.

In StartProcesses, I define a multiprocessing.Manager object to collect the
list of PIDs, what is done in each process using os.getpid(). StartProcesses
has also the run() method that creates P1,P2,P3 and uses the join() method
to make sure they are all terminated before going further.

Now, cancelling the computation: for processes, I get the Manager object
from StartProcesses and kill each process using os.kill(pid, 3). Not sure
the signal is the good one (I'm on Windows). For the thread, I turn a flag
off so that it knows the computation is interrupted. Basically, it goes to
wx.CallAfter() and there, instead of starting the next thread, I just stop
the computation workflow.

After some 'hit my head against the nearest wall' sessions, it works more or
less: when I cancel the computation, I get all processes (P1, ...) killed
(at least, I hope, don't really know how to check), and only the MainThread
is still alive (I check this by using threading.enumerate()).

Now the problem: if I look at Windows Task manager -> Processes, I have only
one python.exe process when I start the application. When I launch the
computation, I get another one, than a few more when it gets to running P1,
P2, etc. When I cancel the computation, as all processes P1, etc. and the
ComputationThread are terminated, I should get back to only one python
process, but no, there are 2 of them! So, I'd like to know why there is an
extra python process still running, and what it is ??? When I quit the
application, everything is gone (at least), but now, if I do compute/cancel
several times, without quitting the application, everytime one extra python
process is added, so after 3 loops, for example, I have 4 python processes
running.

Sorry, this is a bit complicated, but if anyone has a hint, that would be
great.

I normally use the terminate() method of the multiprocessing Process
class, always from a separate thread (which monitors and eventually
kill the process itself). If you wish to check what and where is still
running, I normally do this check by looping through the processes and
using is_alive() (combined with multiprocessing.active_children()).

Andrea.

"Imagination Is The Only Weapon In The War Against Reality."
http://xoomer.alice.it/infinity77/

···

On 25 May 2012 15:10, Raphael Mayoraz wrote:

3

I use multiprocessing in several applications and have not seen this behavior. That said I have not used multiprocessing.Manager before, but I think Manager itself is a Python process. If so, that might explain your extra process. Also, when I “kill” processes, I use the same method you used for threads: set a flag. With processes, I tend to send a stop message (via Queue or Pipe).

If you can share some code to reproduce the problem, I might be able to help more.

···

On Friday, May 25, 2012 10:40:36 AM UTC-2:30, rapmay wrote:

Guys,

I know this is a topic for the Python list, not the wxPython, but you
guys are so ressourceful that I dare submitting my problem to you.

As in most of the software, an ‘Cancel computation’ button is required
and I have trouble to make it work clean.

Here is how my application works. The Cancel computation button is, of
course, in the main thread.

                      MainThread (mostly the GUI)

                           >

                   ComputationThread1 (threading.Thread)

  at the end of run(), wx.CallAfter(), starts the next thread

                           >

                   ComputationThread2 (threading.Thread)

                           >

                  StartProcesses (a regular class)

                  >        >        >

                 P1       P2       P3  (those are

multiprocessing.Process objects)

 etc.

In StartProcesses, I define a multiprocessing.Manager object to collect
the list of PIDs, what is done in each process using os.getpid().
StartProcesses has also the run() method that creates P1,P2,P3 and uses
the join() method to make sure they are all terminated before going further.

Now, cancelling the computation: for processes, I get the Manager object
from StartProcesses and kill each process using os.kill(pid, 3). Not
sure the signal is the good one (I’m on Windows). For the thread, I turn
a flag off so that it knows the computation is interrupted. Basically,
it goes to wx.CallAfter() and there, instead of starting the next
thread, I just stop the computation workflow.

After some ‘hit my head against the nearest wall’ sessions, it works
more or less: when I cancel the computation, I get all processes (P1,
…) killed (at least, I hope, don’t really know how to check), and only
the MainThread is still alive (I check this by using threading.enumerate()).

Now the problem: if I look at Windows Task manager → Processes, I have
only one python.exe process when I start the application. When I launch
the computation, I get another one, than a few more when it gets to
running P1, P2, etc. When I cancel the computation, as all processes P1,
etc. and the ComputationThread are terminated, I should get back to only
one python process, but no, there are 2 of them! So, I’d like to know
why there is an extra python process still running, and what it is ???
When I quit the application, everything is gone (at least), but now, if
I do compute/cancel several times, without quitting the application,
everytime one extra python process is added, so after 3 loops, for
example, I have 4 python processes running.

Sorry, this is a bit complicated, but if anyone has a hint, that would
be great.

Thanks,

Raphael

4

Thanks Andrea and Jay,

Yep, multiprocessing.Manager is a process (what I ignored), and
multiprocessing.active_children() helps a lot figuring everything out.
It looks like everything is clean now (famous last word).

Raphael

···

On 5/25/2012 3:23 PM, Andrea Gavana wrote:

Hi,

On 25 May 2012 15:10, Raphael Mayoraz wrote:

Guys,

I know this is a topic for the Python list, not the wxPython, but you guys
are so ressourceful that I dare submitting my problem to you.
As in most of the software, an 'Cancel computation' button is required and I
have trouble to make it work clean.

Here is how my application works. The Cancel computation button is, of
course, in the main thread.

                         MainThread (mostly the GUI)
                              >
                      ComputationThread1 (threading.Thread)
     at the end of run(), wx.CallAfter(), starts the next thread
                              >
                      ComputationThread2 (threading.Thread)
                              >
                     StartProcesses (a regular class)
                     > > >
                    P1 P2 P3 (those are multiprocessing.Process
objects)

    etc.

In StartProcesses, I define a multiprocessing.Manager object to collect the
list of PIDs, what is done in each process using os.getpid(). StartProcesses
has also the run() method that creates P1,P2,P3 and uses the join() method
to make sure they are all terminated before going further.

Now, cancelling the computation: for processes, I get the Manager object
from StartProcesses and kill each process using os.kill(pid, 3). Not sure
the signal is the good one (I'm on Windows). For the thread, I turn a flag
off so that it knows the computation is interrupted. Basically, it goes to
wx.CallAfter() and there, instead of starting the next thread, I just stop
the computation workflow.

After some 'hit my head against the nearest wall' sessions, it works more or
less: when I cancel the computation, I get all processes (P1, ...) killed
(at least, I hope, don't really know how to check), and only the MainThread
is still alive (I check this by using threading.enumerate()).

Now the problem: if I look at Windows Task manager -> Processes, I have only
one python.exe process when I start the application. When I launch the
computation, I get another one, than a few more when it gets to running P1,
P2, etc. When I cancel the computation, as all processes P1, etc. and the
ComputationThread are terminated, I should get back to only one python
process, but no, there are 2 of them! So, I'd like to know why there is an
extra python process still running, and what it is ??? When I quit the
application, everything is gone (at least), but now, if I do compute/cancel
several times, without quitting the application, everytime one extra python
process is added, so after 3 loops, for example, I have 4 python processes
running.

Sorry, this is a bit complicated, but if anyone has a hint, that would be
great.

I normally use the terminate() method of the multiprocessing Process
class, always from a separate thread (which monitors and eventually
kill the process itself). If you wish to check what and where is still
running, I normally do this check by looping through the processes and
using is_alive() (combined with multiprocessing.active_children()).

Andrea.

"Imagination Is The Only Weapon In The War Against Reality."
http://xoomer.alice.it/infinity77/